If you have GNU Parallel installed you can do:
doit() {
file1="$1"
file2="$2"
output="$3"
awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
/:/ || !NF{print; next}
{print $1, $2*a[$1], $2*b[$1]}' "$file2" "$file1" > "$output"
}
export -f doit
# If all filenames fit on a command line:
parallel --xapply doit {1} {2} {1/.}{2/.}.out ::: ../mo/?*e.log ::: ../excited/?*d.log
# With newer versions you can do:
parallel doit {1} {2} {1/.}{2/.}.out ::: ../mo/?*e.log :::+ ../excited/?*d.log
# If you do not like the {/.} you can do:
parallel doit {1} '{= s/e.log/d.log/;s:/mo/:/excited/:; =}' '{=s/.log/.out/;s:^../mo/::;=}' ::: ../mo/?*e.log
# If all the files do not fit on the command line (because you have thousands):
finda() { find ../mo/ -name '*e.log'; }
findb() { find ../excited/ -name '*d.log'; }
parallel --xapply doit {1} {2} {1/.}{2/.}.out :::: <(finda) <(findb)
parallel doit {1} {2} {1/.}{2/.}.out :::: <(finda) ::::+ <(findb)
finda | parallel doit {1} '{= s/e.log/d.log/;s:/mo/:/excited/:; =}' '{=s/.log/.out/;s:^../mo/::;=}'
It will run one job per core. If you prefer one job at a time, replace parallel
with parallel -j1
.
GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for
loop.
If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:
GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:
Installation
If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:
(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash
For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README
Learn more
See more examples: http://www.gnu.org/software/parallel/man.html
Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html
Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel
e.log
and-d.log
files, or only for specific pairs? if the latter, how are the filenames related?