I'd like to combine a bunch of language model (LM) count files using SRILM's ngram-merge program. With that program, I could combine a whole directory of count files, e.g. ngram-merge -write combined_file -- folder/*. However, with my amount of data it would run for days, so I'd love to merge the files in parallel!

The script below does basically the following:

  1. It splits the files in the directory into two equally sized sets (if the number of files is odd, two files are merged before the sets will be build)
  2. It loops through the sets and merges two files together, whereby the new file is written into a new subdirectory (this should be done in parallel)
  3. It examines whether there is only one file in the new subdirectory; if not, 1. starts again in the newly created subdirectory

The script works, unfortunately, though, the ngram-merge commands are not compiled in parallel. Can you fix that? Furthermore, the folder structure that is created on the fly is kinda ugly. And I'm also not a shell-expert. So I'll be thankful for every remark that makes the whole thing more elegant!!! Thx :-)


# Get arguments
# Count number of files
number="$(ls -1 $indir | wc -l)"
# Determine number of cores to be used in parallel

# While more than one file, combine files
while [ "$number" -gt 1 ]; do

    # determine splitpoint
    # Determine whether number of files is odd
    if [ $((number%2)) -eq 1 ]
        # if it is odd, combine first and last file and rm last file
            first="$indir$(ls -1 $indir | head -1)"
            last="$indir$(ls -1 $indir | tail -1)"
            /vol/customopt/lamachine.stable/bin/ngram-merge -write $new -- $first $last && rm -r $first $last

    # Determine first half of files and second half
    set1="$(ls -1 $indir | head -$split)"
    set2="$(ls -1 $indir | head -$((split*2)) | tail -$split)"
    # Make new dir
    mkdir $newdir

    # Paralelly combine files pairwise and save output to new dir
    for i in $(seq 1 $split); do
        file1="$indir$(echo $set1 | cut -d " " -f $i)"
        file2="$indir$(echo $set2 | cut -d " " -f $i)"
        /vol/customopt/lamachine.stable/bin/ngram-merge -write $newfile -- $file1 $file2 && rm -r $file $file2
        ((i=i%N)); ((i++==0)) && wait

    # Set indir = newdir and recalculate number of files
    number="$(ls -1 $indir | wc -l)"


I do not know ngram-merge so I use cat:

n=$(ls | wc -l)
while [ $n -gt 1 ]; do
  parallel -N2 '[ -z "{2}" ] || (cat {1} {2} > '$n'.{#} && rm -r {} )' ::: *;
  n=$(ls | wc -l);

But it probably looks like this:

n=$(ls | wc -l)
while [ $n -gt 1 ]; do
  parallel -N2 '[ -z "{2}" ] || ( /vol/customopt/lamachine.stable/bin/ngram-merge -write '$n'.{#} -- {1} {2} && rm -r {} )' ::: *;
  n=$(ls | wc -l)
  • great, thanks! Works and it is much more elegant than what I proposed :-)
    – hyhno01
    Mar 27 '20 at 17:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.