I have a directory that contains 400 million files. Using find, I created a list of all the files, which looks like this:
/output/custom/31/7/31767937 /output/custom/31/7/317537a /output/custom/31/7/317537 /output/custom/31/7/317ab /output/custom/31/7/317bo /output/custom/31/7/317je /output/custom/31/7/317ma /output/custom/31/7/31763
I then split the file into 20 different files, and ran a script to create 20 different tarballs:
for i in $(ls x*) do tar -cf /tar/$i.tar -T $i & done
The input files are on a different drive than the
/tar mount point. The script has now been running for 2 days, and it's about 1/4 of the way done. I'll probably just leave it running at this point. However, for future reference, I'm wondering if there's a better way to do this than using
My end goal here is to move these tarballs to 20 different servers, untar them and run some scripts on the files. Oh, and since I then have the tarballs I'll be putting them on S3 storage too.