I am using awk to do match two files and then multiply the elements of file2 and file1.

 awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
      /:/ || !NF{print; next}
     {print $1, $2*a[$1], $2*b[$1]}' file2 file1 > output

This script only process two input files and produce one output file.

I want to do a loop to use this script for many (thousands ) files. I try to do:

for file1 in ../mo/*e.log | 
for file2 in ../excited/*-d.log;   do
awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
     /:/ || !NF{print; next}
     {print $1, $2*a[$1], $2*b[$1]}' "$file1" "$file2" > "${file1%e.log}f.log"

The files are related, so are like 0001e.log and 0001-d.log, 0002e.log and 0002-d.log, 0002e.log and 0002-d.log ... The expected output could be 0001f.log , 0002f.log , 0003f.log ...

But with no success. Any ideas?

  • Please show input and expected output. – 123 Jun 9 '16 at 14:17
  • 1
    What does "with no success" mean, exactly? You mention "match" in the first sentence but I see no evidence of that in your code: do you want to run the awk command for all NxM combinations of e.log and -d.log files, or only for specific pairs? if the latter, how are the filenames related? – steeldriver Jun 9 '16 at 14:29
  • @steeldriver, I want to run the awk command for specific pairs. The filenames are related, so the names are 0001e.log and 0001-d.log , 0002e.log and 0002-d.log , 0003e.log and 0003-d.log .... – alloppp Jun 9 '16 at 15:08
  • Then you need only a single loop - please see Stéphane Chazelas' answer – steeldriver Jun 9 '16 at 15:27

Maybe you want:

set ../mo/*e.log
for file2 in ../excited/*-d.log; do
  file1=$1; shift
  awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
       /:/ || !NF{print; next}
       {print $1, $2*a[$1], $2*b[$1]}' "$file1" "$file2" > "${file1%e.log}f.log"

Or with zsh:

for file1 file2 (${file1s:^file2s}) {
  awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
       /:/ || !NF{print; next}
       {print $1, $2*a[$1], $2*b[$1]}' "$file1" "$file2" > "${file1%e.log}f.log"

Above, we've got 2 sorted lists of file names and we go through both lists in parallel. If the base name of the files in mo and in excited are to be matched instead, you could do:

for file1 in ../mo/*e.log; do
  [ -f "$file2" ] || continue
  awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
       /:/ || !NF{print; next}
       {print $1, $2*a[$1], $2*b[$1]}' "$file1" "$file2" > "${file1%e.log}f.log"
| improve this answer | |
  • The last aswer has an error when I tried to test it. line 2: syntax error near unexpected token `basename=${file1%e.log}' – alloppp Jun 9 '16 at 17:27
  • @allopp, there was a missing do. – Stéphane Chazelas Jun 9 '16 at 17:33
  • I dont know but now the last script works but not output anything? – alloppp Jun 9 '16 at 17:44
  • 1
    @allopp, another obvious mistake. There may be more as I've not tested the solution. You know you're allowed to try and understand and debug the answers. That's not a free solution spoon-feeding service. – Stéphane Chazelas Jun 9 '16 at 18:26

Try paste file1 file2 | tr '\t' '*' | bc > output.

Then for the big loop (with bash), which lines up files from ../mo/, ../excited/, and outputs products to f series of numbered files in current directory:

for f in ../mo/*e.log; do
    paste $f ${g/e.log/-d.log} | tr '\t' '*' | bc > ${o/e.log/f.log} 

Demo, (with bashisms), print the squares of 1-5:

paste <(seq 5) <(seq 5) | tr '\t' '*' | bc


| improve this answer | |

If you have GNU Parallel installed you can do:

doit() {
  awk 'NR == FNR{a[$1]=$2; b[$1]=$3; next}
      /:/ || !NF{print; next}
     {print $1, $2*a[$1], $2*b[$1]}' "$file2" "$file1" > "$output"
export -f doit

# If all filenames fit on a command line:
parallel --xapply doit {1} {2} {1/.}{2/.}.out ::: ../mo/?*e.log ::: ../excited/?*d.log
# With newer versions you can do:
parallel  doit {1} {2} {1/.}{2/.}.out ::: ../mo/?*e.log :::+ ../excited/?*d.log

# If you do not like the {/.} you can do:
parallel doit {1} '{= s/e.log/d.log/;s:/mo/:/excited/:; =}' '{=s/.log/.out/;s:^../mo/::;=}' ::: ../mo/?*e.log

# If all the files do not fit on the command line (because you have thousands):
finda() { find ../mo/ -name '*e.log'; }
findb() { find ../excited/ -name '*d.log'; }

parallel --xapply doit {1} {2} {1/.}{2/.}.out :::: <(finda) <(findb)
parallel doit {1} {2} {1/.}{2/.}.out :::: <(finda) ::::+ <(findb)
finda | parallel doit {1} '{= s/e.log/d.log/;s:/mo/:/excited/:; =}' '{=s/.log/.out/;s:^../mo/::;=}'

It will run one job per core. If you prefer one job at a time, replace parallel with parallel -j1.

GNU Parallel is a general parallelizer and makes is easy to run jobs in parallel on the same machine or on multiple machines you have ssh access to. It can often replace a for loop.

If you have 32 different jobs you want to run on 4 CPUs, a straight forward way to parallelize is to run 8 jobs on each CPU:

Simple scheduling

GNU Parallel instead spawns a new process when one finishes - keeping the CPUs active and thus saving time:

GNU Parallel scheduling


If GNU Parallel is not packaged for your distribution, you can do a personal installation, which does not require root access. It can be done in 10 seconds by doing this:

(wget -O - pi.dk/3 || curl pi.dk/3/ || fetch -o - http://pi.dk/3) | bash

For other installation options see http://git.savannah.gnu.org/cgit/parallel.git/tree/README

Learn more

See more examples: http://www.gnu.org/software/parallel/man.html

Watch the intro videos: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial: http://www.gnu.org/software/parallel/parallel_tutorial.html

Sign up for the email list to get support: https://lists.gnu.org/mailman/listinfo/parallel

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.