How to copy Big files on other Linux Server Independent of TTY

Question

Actually i was trying to copy all the files in another director. Total size of the file is 188GB so its going to take a time.

Im trying to copy for Linux server A to B using this command

nohup scp Files-from-Server-A root@xxx.xxx.xxx.xxx:tmp &

but as soon i close the terminal it stops copying the files but the process is still shown in process manager.

Answer 1

scp does detect whether it's got a controlling tty. If you run it as a foreground process it will show a progress bar, but if you background it, the progress bar disappears, so there are some internal checks going on. I would do as @peterph says and start the transfer in either a tmux or screen session. I'm not sure of the implications of using nohup, but just keep a controlling terminal active.

As the file is that big, if it's not compressed, I'd also add compression to the transfer with scp's -C option.

If it's a directory, and you've been retrying the same command quite often, you should probably use rsync, which first checks if files differ (with md5 checksums) before transferring them. You can also resume big file transfers with the --partial option.

rsync -avz --partial --progress --rsh=ssh Files-from-Server-A [...] user@B:/tmp/

Answer 2

General advice is to use terminal multiplexer (e.g. tmux or screen) and run the copy inside, since that gives one much flexibility. You can detach/reattach as necessary to check how the operation proceeds.

These programs create "virtual" (headless) terminals which are displayed on/take input from a "real" terminal (linux console, X terminal emulators, ssh connection...). As the virtual terminal is not affected by interruptions in the real terminals (the terminal multiplexer just stops displaying output when it detects the IO device being closed and keeps storing output into it's internal buffer), the applications running inside are shielded from SIGHUP. As a bonus, they usually offer at least some basic text-mod windowing capabilities.

I'm not sure about tmux (which is actively developed), but GNU screen should be included in standard RHEL repository.

Answer 3

lftp (a general purpose terminal file transfer program) supports sftp (not scp, but if you have scp, you have ssh and most likely it supports sftp as well), and it can put transfers in background, automatically reconnect and resume the transfer (after a customisable delay) upon failure and so on.

Answer 4

Can't you use rsync? Such a big transfer is sure to break somewhere, and I'm not sure the mechanisms proposed are able to continue from where it broke off.