I regularly need to transfer millions of small files (small images, txt, json) with average of 5-50k per file between servers or to aws s3.
Is there a faster to merge them into a single file to optimize transfer speed other than zip/tar -cf?
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.Sign up to join this community
Developing idea from other answer you can send information via pipe and even do not create locally a
tar file for example. Command will be something like:
tar cf - * | aws s3 cp - s3://some-bucket/archive.tar
This command have advantage that will run
aws command in parallel. You can even add compression (which will execute again in parallel)
tar cf - * | gzip -c | aws s3 cp - s3://some-bucket/archive.tar.gz
For simplify the operation you can use top level directory for the files and not use
tar cf - top_level_directory | aws s3 cp - s3://some-bucket/archive.tar tar cf - top_level_directory | gzip -c | aws s3 cp - s3://some-bucket/archive.tar.gz
Inspired by other answer you can use
cpio. Seems to be faster and produce smaller files:
ls |cpio -o |gzip -c | aws s3 cp - s3://some-bucket/archive.cpio.gz
One is to never create an intermediate file, as suggested in the other answers. This cuts out local IO, though it makes resuming a partial upload impossible.
There are other options though to improve things further:
tarand what you’re describing. ZSTD is not as fast as LZ4, but will get you compression ratios similar to GZip in less time. Irrespective of the choice, this will cut down, possibly significantly, on the total data to transfer.
taris not exactly a space-efficient archive format. This usually doesn’t matter much, but if you’re dealing with millions of very small files, the overhead is actually pretty significant.
cpiostill has a nontrivial amount of overhead, but it’s less than
tardoes, so in theory using
cpiohere should result in a nontrivial reduction in the amount of data to transfer.
It might be worth it to investigate the rsync command as what you want to do is transfer the files.
--- Added after comments.
This is an XY question: "It takes long time to transfer files so I want to compress them more?". I believe a better question would be: "It takes a long time to transfer the files, how can I make it happen faster?". Rsync is an answer to the second question.
Note that rsync creates and copies the files, it does not create an archive you later need to decompress (which would add to total time as well). As it copies, basically, "changed" or "new" files it can be set to run nearly continuously (say every hour) and will have no problems restarting after network failures. It pays to think a bit about directory structure though.
The exact options to use for rsync in different situations is dependant on the relative speeds of network compared to CPU speed. One important thing to note is that if you select compression it is done simultaneously with transfer. Compared to zipping or similar where you first compress, then transfer, then decompress. This can save a lot of "total time" even when maybe the compression is skipped. (Compression in my experience saves very little time in rsync when sending lots of small files).
Rsync is available on most if not all modern systems, including servers and aws s3 according to my understanding, but I may be wrong.
What is your storage?
This answer shows that on a single spindle, sorting by inode and running jobs in parallel will get the fastest results: