4

I'm trying to move parts of a large directory (~40 GiB and ~8 million files) across multiple machines via Amazon S3, and due to needing to preserve symlinks I am tarring up the directory and then uploading the resultant file, rather than syncing directly to S3.

Most of the files are already compressed, so I'm not compressing the archive with gzip or bzip. My command is along the lines of

tar --create --exclude='*.large-files' --exclude='unimportant-directory-with-many-files' --file /tmp/archive.tar /directory/to/archive

While running this, I've noticed that tar only appears to use one core on the eight-core machine. My impression, based on the pegging of that core, the low load average (~1), and the stats I'm seeing from iostat is that this operation is actually cpu-bound, rather than disk-bound, as I'd expect. Since it's slow (~90 minutes), I'm interested in trying to parallelize tar to make use of the additional cores.

Other questions on this topic either focus on compression or create multiple archives (which, due to the directory structure, is not easy in my situation). It seems most people forget that you can even create a tarball without compressing it.

  • Not with standard tar. tar stands for Tape ARchiver, once upon a time it was used to write backups on tapes (it still can do that, if you have the hardware). The physical format was exactly the tar format (or rather pax). Since you can't write things in parallel on a piece of tape, tar can't archive files in parallel. Maybe newer incarnations of it can though. On a side note: the format has rather low limits on paths length. Make sure you don't run into those before trying to archive 8 mil. files. – Satō Katsura Jul 8 '16 at 20:55
  • Actually, a tar program could be written to both read/write in parallel, since the output archive is not compressed, and because the size of each file (and header) can be computed directly (allowing a program to write a different extents). There aren't a lot of applications which need that, however. – Thomas Dickey Jul 8 '16 at 21:02
  • @ThomasDickey That's why I said that perhaps newer incarnations can do it. I'm not aware of any that actually does though. – Satō Katsura Jul 8 '16 at 21:09
6

Because of the nature of a tar archive which sequentially stores the files in the output, there is no way to parallelize the process unless you make more than one archive.

Note that the bottleneck of the operation would likely be the hard drive. For that reason, even if you did split the task in two or more processes, it would not go faster unless they operate on different drives.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.