I have some experience with Inix terminals from science internships I've been part of, mainly using a few utilities like grep, awk, and sed but there is one thing I've been trying to figure out for a while that would really make me far more efficient with the number crunching I have to do.

I have a script run.awk that performs some manipulations on a large of collection of massive text files. As it is it will take the file chloride.out, extract data from it and write chloride.cm .

Is there anyway I can make this script take in *.out and write *.cm files based off the initial wildcard phrase in shell?

The amount of scripts I've written for processing large amounts of data that I've had to do a hundred+ iterations of is just annoying.

Ideally I would like to know if there is a way that I can do this for all my scripts with something via the shell. If it can't be automated in the shell or an equivalent can I least automate my awk scripts in a similar manner as I've described?


You most certainly can make awk deal with multiple files via wildcards. One suggestion would be to leave the run.awk as a generic "function" that takes a single file in and produces a single output file, and then call it from another script which could then take care of assimilating the input and output files.


This would be a Bash script, we can call it, awk_runner.bash.


for ifname in *.out; do 
  printf "IN: %s, OUT: %s\n" $ifname $ofname
  printf "running run.awk with %s & %s\n\n" $ifname $ofname

  run.awk $ifname $ofname

Sample run

I made a example directory with some test files in it.

$ touch file{1..4}.out

This resulted in 4 files being made:

$ ls -1

Now we run our script:

$ ./awk_runner.bash
IN: file1.out, OUT: file1.cm
running run.awk with file1.out & file1.cm

IN: file2.out, OUT: file2.cm
running run.awk with file2.out & file2.cm

IN: file3.out, OUT: file3.cm
running run.awk with file3.out & file3.cm

IN: file4.out, OUT: file4.cm
running run.awk with file4.out & file4.cm

After each line that starts with, "running..." our script could run from here.

Files in a list

Say instead of using the wildcard, *.out we instead had a file with a list of filenames in it, say:

$ cat filelist.txt 

We could use this modified version of our script which would use a while loop instead of a for loop. Now let's call this variant of the script, awk_file_runner.bash:


while read ifname; do 
  printf "IN: %s, OUT: %s\n" $ifname $ofname
  printf "running run.awk with %s & %s\n\n" $ifname $ofname

  run.awk $ifname $ofname
done < filelist.txt

This version of the script reads the input from the file, filelist.txt:

done < filelist.txt

Then for each turn of the while loop, we're using the read command to read in a line from the input file.

while read ifname; do

It then performs everything in the same way as the first script where it will run the awk script run.awk as it loops through each line of the file.

  • is there anyway i can also abstractualize this one step further, and feed this bash script the scripts from a file. I don't need it for this application but it might come in handy somewhere. Also I would delete that arrow right by run.awk right?
    – user507974
    Oct 1 '13 at 1:24
  • @user507974 - yes I removed the comment from the script, sorry if that was confusing you. It now only shows what you'd need. Of course you could run this using names from a file instead. I'll add that as a 2nd example.
    – slm
    Oct 1 '13 at 1:28
  • @user507974 - see updates, let me know if you have any other questions about this answer.
    – slm
    Oct 1 '13 at 1:39

Rather than writing a shell wrapper and spawning a new awk instance for every file you process, you can do this in awk directly. If you already have an awk script, you can access the current file using the FILENAME variable. So, if you run awk 'some commands' file1 file2, you can tell if you're working with file1 or file2 using FILENAME. You can also use > on print/printf in awk. So, if you have an awk script like

/pattern/{ print $1,$3 }

you could easily do

/pattern/{ print $1,$3 > FILENAME".processed" }

or use FNR=1 to tell when you're in a new file, and create a variable to do more complex manipulation on the filename. Like replacing a .in extension with .out, as in

sauer@humpy:/tmp$ grep . file*.in
sauer@humpy:/tmp$ awk 'FNR=1{out=FILENAME;sub("\.in$",".out",out)} {print "processed"$0 > out}' file*.in
sauer@humpy:/tmp$ grep . file*.out

I'm using grep . to show the filename and the contents of multiple files here, which is also a fun trick. But the important thing is setting the value of the out variable to a modified version of FILENAME when FNR changes to 1 (so we're on line 1 of the file), and then redirecting all prints to out. Note that this is mildly dangerous, in that failure to match the extension will result in no substitution, leading to overwriting your input files. So, it'd be good to add in a fail-safe check to make sure that out != FILENAME or something like that as well. That's left as an exercise for the reader. ;)

If you need a file containing a list of filenames, it's easiest to just run it like

awkscript $(< /path/to/filename_list_file )

Which takes the contents of the filename_list_file and puts it on the command line.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.