Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a requirement to merge chunks of text files in a directory to one or more files and then zip the merged file.

For ex: If i have half a million text files in a directory, say A. I want to merge sets of 50,000 files into 1 i.e. I will have 10 merged files each having the content of 50,000 files. Finally, I want to zip each merged file, i.e. there will be totally 10 zip files at the end of this process.

I tried doing this in shell script. I am unable to find a solution.

Can someone let me know the most efficient way to achieve this using a shell script?

share|improve this question

2 Answers 2

As I can see you have just one dir with huge set of files, therefore you need to split filelist into small sets, than to concatenate them , then zip and if necessary to delete small files.

I can suggest only the following way:

cd /to/dir/with/many/files/
mkdir ../tmp
find . | split -l 50000 ../tmp/x
for i in ../tmp/x* ; do 
    while read filename ; do
        cat "$filename" >> "${i}.text" ;
        # rm "$filename" # uncomment it only if you need to delete small files
    done <"$i"
    zip "${i}.zip"  "${i}.text"
    # rm "${i}.text" # uncomment it only if you need to delete result file
done
share|improve this answer

Here are two solutions, one where n files are added to a single archive and one where all the files are first concatenated and then zipped.

Common steps to both approaches are:

# Generate a list of files to be zipped
find . -type f > filelist

# Loop to process n files stepwise
n=50000
fileno=1

for i in $(seq 1 $n $(wc -l < filelist)); do
  # compression code goes here, see below
done

Zip files individually

# automatic name generation
zipfile=$(printf "%04d" $((fileno++)))

# extract lines $i to $i+$n-1 from filelist 
sed -n "$i,$((i+n-1))p" filelist | zip $zipfile -@

Concatenate and zip

If you wanted to do this with gzip (and other compressors) it would be quite simple:

zipfile=$(printf "%04d" $((fileno++)))
sed -n "$i,$((i+n-1))p" filelist | xargs cat | gzip > $zipfile.gz

As zip doesn't support this mode (at least not the one I have), you need a different approach. zip supports named pipes (-FI) where the file inside the archive gets the name of the named pipe, so doing something like this inside the loop should work:

zipfile=$(printf "%04d" $((fileno++)))
mkfifo $zipfile
zip -FI $zipfile $zipfile &
sed -n "$i,$((i+n-1))p" filelist | xargs cat > $zipfile
rm $zipfile
share|improve this answer
    
Thanks for the quick response.filename can have a standard prefix, say 1ei. I want to first merge the sets of 50000 files into one text file and then zip it. –  nka Aug 22 '12 at 14:59
    
Thanks for the response..I made some modifications to this script.. for i in $(seq 1 2 10); do sed -n "$i,$((i+2 - 1))p" filelist | xargs cat > $i.kkk | zip $i.zip $i.kkk; done In this code, I am able to merge the files, but when I try to zip, it is creating issue.. –  nka Aug 23 '12 at 6:53
    
I would recommend sticking with the zip fname -@ which will add filenames on stdin to fname.zip. I added some better zip filename generation to the answer. Also see here for a similar question. –  Thor Aug 23 '12 at 8:22
    
Thanks..This program zips 50000 txt files in one zip...But I want to first merge all those 50000 txt files into one txt file and then create a zip for that one txt file...Is it possible to achieve this in a single piped command? –  nka Aug 23 '12 at 8:53
    
Ah I thought that was what you wanted to do, I'll add an update with concatenation and zipping. –  Thor Aug 23 '12 at 10:10

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.