-2

I want to transfer ~300G files from a laptop to other in the same local wifi network, for back up purpose. (I am running out of the space of my only external hard drive, and can't afford anything. Someone was kind enough to leave an old laptop in a dumpster for me to uncover.) Both now run Lubuntu.

What is the fastest and reliable way to transfer the files? By rsync, scp, or some other commands? Could you also give some concrete commands to transfer for example two directories (and the files under them)?

How is the transfer speed of the command you will use compared to using an external hard drive to physcially connect in turn to the computers via a USB cable and perform the transfer indirectly, If there is another external hard drive available in a dumpster in the future ?

Thanks.

3

On machine receiver.example.com:

cd /destination
socat -u tcp-listen:33333,reuseaddr - | gunzip | pv -trab | bsdtar xpSf -

Then on machine sender.example.com:

cd /source
bsdtar cf - . | pigz -3 | socat -u - tcp:receiver.example.com:33333

Transfer speed will depend on how fast and reliable your WiFi connection is and how compressible your data is (possibly how fast the CPU is if the laptop is really old and the compression ends up being the bottleneck).

You may need to install the bsdtar, pigz, socat and pv packages.

  • 1
    Is it possible to recover/continue if connection is disrupted during transfer? – sebasth Feb 18 at 14:50
  • 1
    @sebasth, not really, you'd want to resume with rsync. – Stéphane Chazelas Feb 18 at 14:52
  • 2
    @sebasth, if the wifi speed ends up being the bottleneck (likely), then rsync with the right --compress options could give you approaching performances. – Stéphane Chazelas Feb 18 at 14:54
  • 1
    Sometimes it's worth breaking the work up into smaller chunks (instead of transferring all the files in one go, do it in by subdirectory for example) so on failure you don't have to retransmit as much. – Stephen Harris Feb 18 at 14:56
3

rsync is rather standard tool for transferring files. One advantage of rynsc is also that it can resume interrupted transfers.

It use ssh to transfer the data over network:

rsync [OPTION...] SRC... [USER@]HOST:DEST

For example (-a archive, -z compress, -v verbose):

rsync -avz /local/path user@remote:/remote/path

You can also use rsync for copying to local path (external HDD, NFS mount, etc).

2

1> go to MediaMarkt
2> buy a 500GB USB Hard drive
3> transfer your data onto that HD
4> plug the HD to destination machine
5> transfer data on destination machine

(considering an HDD) USB2 ~25MiB/s, USB3 ~80MiB/s,
ethernet 100Mbps ~12MiB/s, ethernet 1Gbps ~120MiB/s

  • Thanks. I am not sure how to understand "ethernet 100MBps ~12MiB/s, ethernet 1Gbps ~120MiB/s". Are both for ethernet? Do you mean a range by "100MBps ~12MiB/s"? Are MBps and MiB/s the same unit? – Tim Feb 18 at 14:44
  • No those are not. 1 Mb = megaBIT = 1 000 000 bit, 1 MiB MebiByte = 1024 KiByte = 1 048 576 Byte = 8 388 608 bit, 1 MB = 1 MegaByte = 1000 kByte = 1 000 000 Byte = 8 000 000 bit (1 byte=8bit, M = 10^6, Mi = 2^20) – DDS Feb 18 at 14:50
1

I use these two netcat (nc) scripts to send/receive a directory tree

  • The sender needs to know the receiver IP
  • The receiver needs to be run from the parent directory where the received directory will be opened up
  • The receiver needs to be started before sender
  • Assumed that the two linux (or netcat having) computers are on the same net
  • And in walking (or shout-across-the-hallway) distance. Adjust the 30 sec (-w 30) part of the receiver if you need more time to activate the sender
  • If you prefer to send/receive a file rather than a directory change the tar pipe to a file redirect

The receiver

$ cat recvnc 
# Receives a directory
# Should be run from parent directory
nc -vv -w 30 -p 5600 -l |tar zxv
$

The sender

$ cat sendnc
# $1 is directory to send $2 is other IP

if [ "$#" -ne 2 ] && ! [ -d "$1" ]; then
  echo "Usage: $0 directorytoSend otherIP"
  exit 1
fi
tar zcv $1 |nc -vv -n -w 2 $2 5600
$

Speed

  • I believe you will get the best speed if you plug in the ethernet cable back-to-back between your two machines.
  • They will of course need to be given IPs
  • …that are different!

Through the router (etc) also works but you can expect the router to be slower than copper!

Wifi will also work and be slower still

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.