I have a directory containing million xml files. I want to create multiple zip files containing n number of xml files. For ex: if n = 10000, then I want to create 100 zip files i.e. each zip file will have only 10000 xml files.

What is the most efficient way to create these zip files? I don't want this process to take more time since I have to repeat this operation for multiple folders having millions of xml files.


This may be usefull :

$ zip all.zip *.xml  ## create a big archive.

Find the archive's total size:

$ ls -lh all.zip

Now, split the big archive into small archives, as you need:

$ split -b 5M all.zip ## creates 5 MB zips

Note: extracting the archives does not work, unless you join them back:

cat x* > oldbigzip.
| improve this answer | |

I would do it like this:

# Create file list
find . -type f -name '*.xml' > filelist


# Loop through list, extracting n names each time
for i in $(seq 1 $n $(wc -l < filelist)); do
  zipfile="zipfile$(printf "%04d" $((fileno++)))"
  sed -n "$i,$((i+n-1))p" filelist | zip $zipfile -@
| improve this answer | |
  • I tried this script. It works fast in local unix drive. But, the same script is too slow in a NFS drive. Is there any way to make it faster in NFS drive? – user1617267 Aug 31 '12 at 7:14
  • This is as fast as it can get, the NFS link is the bottleneck. Run the script on the NFS server or get a faster network link. – Thor Aug 31 '12 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.