I'd like to combine a bunch of language model (LM) count files using SRILM's
ngram-merge program. With that program, I could combine a whole directory of count files, e.g.
ngram-merge -write combined_file -- folder/*. However, with my amount of data it would run for days, so I'd love to merge the files in parallel!
The script below does basically the following:
- It splits the files in the directory into two equally sized sets (if the number of files is odd, two files are merged before the sets will be build)
- It loops through the sets and merges two files together, whereby the new file is written into a new subdirectory (this should be done in parallel)
- It examines whether there is only one file in the new subdirectory; if not, 1. starts again in the newly created subdirectory
The script works, unfortunately, though, the
ngram-merge commands are not compiled in parallel. Can you fix that? Furthermore, the folder structure that is created on the fly is kinda ugly. And I'm also not a shell-expert. So I'll be thankful for every remark that makes the whole thing more elegant!!! Thx :-)
#!/bin/bash # Get arguments indir=$1 # Count number of files number="$(ls -1 $indir | wc -l)" # Determine number of cores to be used in parallel N=40 # While more than one file, combine files while [ "$number" -gt 1 ]; do # determine splitpoint split="$((number/2))" # Determine whether number of files is odd if [ $((number%2)) -eq 1 ] # if it is odd, combine first and last file and rm last file then first="$indir$(ls -1 $indir | head -1)" last="$indir$(ls -1 $indir | tail -1)" new="$first""$last" /vol/customopt/lamachine.stable/bin/ngram-merge -write $new -- $first $last && rm -r $first $last fi # Determine first half of files and second half set1="$(ls -1 $indir | head -$split)" set2="$(ls -1 $indir | head -$((split*2)) | tail -$split)" # Make new dir newdir="$indir"merge/ mkdir $newdir # Paralelly combine files pairwise and save output to new dir ( for i in $(seq 1 $split); do file1="$indir$(echo $set1 | cut -d " " -f $i)" file2="$indir$(echo $set2 | cut -d " " -f $i)" newfile="$newdir""$i".counts /vol/customopt/lamachine.stable/bin/ngram-merge -write $newfile -- $file1 $file2 && rm -r $file $file2 ((i=i%N)); ((i++==0)) && wait done ) # Set indir = newdir and recalculate number of files indir=$newdir number="$(ls -1 $indir | wc -l)" done