How to merge chunks of text files in a directory to one or more files?

Question

I have a requirement to merge chunks of text files in a directory to one or more files and then zip the merged file.

For ex: If i have half a million text files in a directory, say A. I want to merge sets of 50,000 files into 1 i.e. I will have 10 merged files each having the content of 50,000 files. Finally, I want to zip each merged file, i.e. there will be totally 10 zip files at the end of this process.

I tried doing this in shell script. I am unable to find a solution.

Can someone let me know the most efficient way to achieve this using a shell script?

Answer 1

As I can see you have just one dir with huge set of files, therefore you need to split filelist into small sets, than to concatenate them , then zip and if necessary to delete small files.

I can suggest only the following way:

cd /to/dir/with/many/files/
mkdir ../tmp
find . | split -l 50000 ../tmp/x
for i in ../tmp/x* ; do 
    while read filename ; do
        cat "$filename" >> "${i}.text" ;
        # rm "$filename" # uncomment it only if you need to delete small files
    done <"$i"
    zip "${i}.zip"  "${i}.text"
    # rm "${i}.text" # uncomment it only if you need to delete result file
done

Answer 2

Here are two solutions, one where n files are added to a single archive and one where all the files are first concatenated and then zipped.

Common steps to both approaches are:

# Generate a list of files to be zipped
find . -type f > filelist

# Loop to process n files stepwise
n=50000
fileno=1

for i in $(seq 1 $n $(wc -l < filelist)); do
  # compression code goes here, see below
done

Zip files individually

# automatic name generation
zipfile=$(printf "%04d" $((fileno++)))

# extract lines $i to $i+$n-1 from filelist 
sed -n "$i,$((i+n-1))p" filelist | zip $zipfile -@

Concatenate and zip

If you wanted to do this with gzip (and other compressors) it would be quite simple:

zipfile=$(printf "%04d" $((fileno++)))
sed -n "$i,$((i+n-1))p" filelist | xargs cat | gzip > $zipfile.gz

As zip doesn't support this mode (at least not the one I have), you need a different approach. zip supports named pipes (-FI) where the file inside the archive gets the name of the named pipe, so doing something like this inside the loop should work:

zipfile=$(printf "%04d" $((fileno++)))
mkfifo $zipfile
zip -FI $zipfile $zipfile &
sed -n "$i,$((i+n-1))p" filelist | xargs cat > $zipfile
rm $zipfile