0

I have a directory that contains 400 million files. Using find, I created a list of all the files, which looks like this:

/output/custom/31/7/31767937
/output/custom/31/7/317537a
/output/custom/31/7/317537
/output/custom/31/7/317ab
/output/custom/31/7/317bo
/output/custom/31/7/317je
/output/custom/31/7/317ma
/output/custom/31/7/31763

I then split the file into 20 different files, and ran a script to create 20 different tarballs:

for i in $(ls x*)
do
    tar -cf /tar/$i.tar -T $i &
done

The input files are on a different drive than the /tar mount point. The script has now been running for 2 days, and it's about 1/4 of the way done. I'll probably just leave it running at this point. However, for future reference, I'm wondering if there's a better way to do this than using tar?

My end goal here is to move these tarballs to 20 different servers, untar them and run some scripts on the files. Oh, and since I then have the tarballs I'll be putting them on S3 storage too.

  • Tar is fine. Running 20 i/o-intensive programs on the same disk(s) probably isn't. With no tars running, look at the output of iostat 5 while gradually starting additional tar processes until one of the disks' throughput tops out. – Mark Plotnick Sep 13 '17 at 14:43
  • Will it be faster if you split it into 200 files instead of 20? – hschou Sep 13 '17 at 14:47
  • 1
    You can gradually reduce the workload on the disk head, without having to restart completely, by using ionice on all the tar processes except one, or even kill -stop on most of the tar processes. Try to ensure only one process is doing i/o on one disk. – meuh Sep 13 '17 at 15:42
  • @MarkPlotnick Here's the thing. To create the files I ran some Java code that I wrote which spawned 20 threads, read some input files and then created the output files. Watching iostat/iotop while running that code, I was consistently getting 150MB/s write speed. Running these tar processes, I'm only getting 100MB/s. Running a single process the other day, to create one tar file, I ended the day with less written to the disk than running all 20 at the same time. If I run a single process, I'm only getting around 20MB/s write speed. – Franz Kafka Sep 13 '17 at 15:48
  • 1
    Never parse the output of ls! If you don't quote its output, your script breaks when any filename contains whitespace; when you do quote it, you can't process more than one file. You can't win. Just use for i in * instead and make it a habit to quote every variable you use,always: tar -cf "/tar/$i.tar" -T "$i" & – n.st Sep 14 '17 at 7:22
2

In terms of ease of transfer, I'd suggest rsync.

Advantages

  • One command to do it all, one command to find them
  • Restartable when the network drops in the middle of the transfer
  • Less effort - you don't need to split/tar then transfer/untar
  • Source server can concentrate on reading files, leaving target servers to deal with writes (fewer potential filesystem IO conflicts updating metadata, etc.)

Disadvantages

  • You discard all the effort you've taken so far
  • It's a different solution to the one you have considered
  • Requires a direct connection between both systems (well, actually it doesn't, but you then need to provide an ssh tunnel)

Proposed solutions

  1. all files in the directory tree

    cd /path/to/top/source/directory
    rsync -avPR -e ssh . remoteuser@remotehost:/path/to/top/destination/directory
    
  2. filenames listed in a file

    cd /path/to/top/source/directory
    rsync -avPR -e ssh --files-from=/path/to/filelist . remoteuser@remotehost:/path/to/top/destination/directory
    

    This second solution requires the file names to be listed relative to the top of the source directory, because the filenames as obtained from the filelist file will be applied in the target filesystem too.

If you're going to do either of these as root you will almost certainly need to copy an ssh certificate across from the source server to the destination, so that you can log in as the root user.

|improve this answer|||||
  • Here is my +1, tar is just a stupid file archive, while it is convenient, they are spending all this crazy disk I/O just to copy the files into a single File. If you are unpacking anyway, and its a single distribution to a specific PC, you way as well just do one copy instead of 3. For larger distribution networks, consider using a clustered/distributed FS – crasic Sep 13 '17 at 21:16
  • The only reason I am tarring these files is because they are going on S3 as well as being copied to different systems. If I was only moving these to different servers, I'd definitely be using rsync. – Franz Kafka Sep 14 '17 at 16:59
  • That's the use case I find best for tar or compressed archives, moving many many files to a remote server. The only use case, otherwise as noted, rsync all the way. – Lizardx Sep 18 '17 at 21:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.