Pairwise merging of files in parallel until only 1 file left over

Question

I'd like to combine a bunch of language model (LM) count files using SRILM's ngram-merge program. With that program, I could combine a whole directory of count files, e.g. ngram-merge -write combined_file -- folder/*. However, with my amount of data it would run for days, so I'd love to merge the files in parallel!

The script below does basically the following:

1. It splits the files in the directory into two equally sized sets (if the number of files is odd, two files are merged before the sets will be build)
2. It loops through the sets and merges two files together, whereby the new file is written into a new subdirectory (this should be done in parallel)
3. It examines whether there is only one file in the new subdirectory; if not, 1. starts again in the newly created subdirectory

The script works, unfortunately, though, the ngram-merge commands are not compiled in parallel. Can you fix that? Furthermore, the folder structure that is created on the fly is kinda ugly. And I'm also not a shell-expert. So I'll be thankful for every remark that makes the whole thing more elegant!!! Thx :-)

#!/bin/bash

# Get arguments
indir=$1
# Count number of files
number="$(ls -1 $indir | wc -l)"
# Determine number of cores to be used in parallel
N=40

# While more than one file, combine files
while [ "$number" -gt 1 ]; do

    # determine splitpoint
    split="$((number/2))"
    # Determine whether number of files is odd
    if [ $((number%2)) -eq 1 ]
        # if it is odd, combine first and last file and rm last file
        then
            first="$indir$(ls -1 $indir | head -1)"
            last="$indir$(ls -1 $indir | tail -1)"
            new="$first""$last"
            /vol/customopt/lamachine.stable/bin/ngram-merge -write $new -- $first $last && rm -r $first $last
    fi

    # Determine first half of files and second half
    set1="$(ls -1 $indir | head -$split)"
    set2="$(ls -1 $indir | head -$((split*2)) | tail -$split)"
    # Make new dir
    newdir="$indir"merge/
    mkdir $newdir

    # Paralelly combine files pairwise and save output to new dir
    (
    for i in $(seq 1 $split); do
        file1="$indir$(echo $set1 | cut -d " " -f $i)"
        file2="$indir$(echo $set2 | cut -d " " -f $i)"
        newfile="$newdir""$i".counts
        /vol/customopt/lamachine.stable/bin/ngram-merge -write $newfile -- $file1 $file2 && rm -r $file $file2
        ((i=i%N)); ((i++==0)) && wait
    done
    )

    # Set indir = newdir and recalculate number of files
    indir=$newdir
    number="$(ls -1 $indir | wc -l)"

done

Answer 1

I do not know ngram-merge so I use cat:

n=$(ls | wc -l)
while [ $n -gt 1 ]; do
  parallel -N2 '[ -z "{2}" ] || (cat {1} {2} > '$n'.{#} && rm -r {} )' ::: *;
  n=$(ls | wc -l);
done

But it probably looks like this:

n=$(ls | wc -l)
while [ $n -gt 1 ]; do
  parallel -N2 '[ -z "{2}" ] || ( /vol/customopt/lamachine.stable/bin/ngram-merge -write '$n'.{#} -- {1} {2} && rm -r {} )' ::: *;
  n=$(ls | wc -l)
done