0

I need to transfer multiple files to a HPC cluster. Currently, I've been using the scp command but it takes about 10 minutes or more to transfer one file (about 1.4GB each). I've seen that using tar would speed up the transfer, however I noticed that compressing the files takes much longer than just transferring the files using scp. The tar command I use is:

tar -cvfz files.tar.gz files

I've also tried using rsync, however I keep getting an error saying rsync is not recognized by bash despite it being installed on the remote server. I'm not too knowledgeable about this stuff so any help or advice would be greatly appreciated!

Improve this question
3
  • 1
    You will need to install rsync on the local system. Following that, add the -z flag to your transfer command to enable compression on the fly. Note also that if you're transferring video files or some other specific file types, they may already be compressed. Asking rsync to compress them will just incur overhead without increasing performance.
    – doneal24
    Sep 26 at 18:26
  • Impossible to give you proper hints w/o knowing more details. What's the type of data in that file? Does it offer itself to compression, or is it already some compressed format, e.g. jpg or mkv? What's the I/O subsystem on the machine you're trying the tar on? Are you both reading and writing to the same disk? What's the network link to the HPC cluster like, what is the line speed? One option w/ tar would be not to create a file locally, but pipe the compressed data over the network and decompress on the far end (whether or not that makes sense depends on the CPU on both sides).
    – tink
    Sep 26 at 19:38
  • Add option -C to your scp command to enable ssh's compression.
    – Cyrus
    Sep 26 at 21:56

1 Answer 1

Reset to default
1

Specifically using tar, you can simply pipe output to ssh instead of storing it on local disk, that should be a bit faster because it streams directly through ssh without waiting for entire tar archive to be built:

tar zcvf - files | ssh user@hostname "tar xvzf - -C /home/user"

If it's still slow, it's likely you'd do better without compression at all:

tar cvf - files | ssh user@hostname "tar xvf - -C /home/user"

This is only useful for one-time moving of files though.

When using scp specifically, it has a flag -C which scp simply passes to ssh, to enable compression.

If you decide to use rsync, which supports partial file transfers and skipping reuploading files that didn't change (which should make things faster over the course of multiple uploads), you should search for --compress, -z in its man page.

Improve this answer
1
  • While usually disk IO (and in this case, network IO) is the limiting factor on speed, CPU can sometimes be a problem with compression - using tar -I pigz .... might help.
    – symcbean
    Sep 27 at 12:58

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .