Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a directory containing million xml files. I want to create multiple zip files containing n number of xml files. For ex: if n = 10000, then I want to create 100 zip files i.e. each zip file will have only 10000 xml files.

What is the most efficient way to create these zip files? I don't want this process to take more time since I have to repeat this operation for multiple folders having millions of xml files.

share|improve this question
1  
What have you tried? –  jordanm Aug 22 '12 at 18:55

2 Answers 2

This may be usefull :

$ zip all.zip *.xml  ## create a big archive.

Find the archive's total size:

$ ls -lh all.zip

Now, split the big archive into small archives, as you need:

$ split -b 5M all.zip ## creates 5 MB zips

Note: extracting the archives does not work, unless you join them back:

cat x* > oldbigzip.
share|improve this answer

I would do it like this:

# Create file list
find . -type f -name '*.xml' > filelist

n=10000
fileno=1

# Loop through list, extracting n names each time
for i in $(seq 1 $n $(wc -l < filelist)); do
  zipfile="zipfile$(printf "%04d" $((fileno++)))"
  sed -n "$i,$((i+n-1))p" filelist | zip $zipfile -@
done
share|improve this answer
    
I tried this script. It works fast in local unix drive. But, the same script is too slow in a NFS drive. Is there any way to make it faster in NFS drive? –  user1617267 Aug 31 '12 at 7:14
    
This is as fast as it can get, the NFS link is the bottleneck. Run the script on the NFS server or get a faster network link. –  Thor Aug 31 '12 at 7:54

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.