Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have some experience with Inix terminals from science internships I've been part of, mainly using a few utilities like grep, awk, and sed but there is one thing I've been trying to figure out for a while that would really make me far more efficient with the number crunching I have to do.

I have a script run.awk that performs some manipulations on a large of collection of massive text files. As it is it will take the file chloride.out, extract data from it and write chloride.cm .

Is there anyway I can make this script take in *.out and write *.cm files based off the initial wildcard phrase in shell?

The amount of scripts I've written for processing large amounts of data that I've had to do a hundred+ iterations of is just annoying.

Ideally I would like to know if there is a way that I can do this for all my scripts with something via the shell. If it can't be automated in the shell or an equivalent can I least automate my awk scripts in a similar manner as I've described?

share|improve this question

2 Answers 2

up vote 5 down vote accepted

You most certainly can make awk deal with multiple files via wildcards. One suggestion would be to leave the run.awk as a generic "function" that takes a single file in and produces a single output file, and then call it from another script which could then take care of assimilating the input and output files.

Example

This would be a Bash script, we can call it, awk_runner.bash.

#!/bin/bash

for ifname in *.out; do 
  ofname=${ifname/.out/.cm}
  printf "IN: %s, OUT: %s\n" $ifname $ofname
  printf "running run.awk with %s & %s\n\n" $ifname $ofname

  run.awk $ifname $ofname
done

Sample run

I made a example directory with some test files in it.

$ touch file{1..4}.out

This resulted in 4 files being made:

$ ls -1
file1.out
file2.out
file3.out
file4.out

Now we run our script:

$ ./awk_runner.bash
IN: file1.out, OUT: file1.cm
running run.awk with file1.out & file1.cm

IN: file2.out, OUT: file2.cm
running run.awk with file2.out & file2.cm

IN: file3.out, OUT: file3.cm
running run.awk with file3.out & file3.cm

IN: file4.out, OUT: file4.cm
running run.awk with file4.out & file4.cm

After each line that starts with, "running..." our script could run from here.

Files in a list

Say instead of using the wildcard, *.out we instead had a file with a list of filenames in it, say:

$ cat filelist.txt 
file1.out
file2.out
file3.out
file4.out

We could use this modified version of our script which would use a while loop instead of a for loop. Now let's call this variant of the script, awk_file_runner.bash:

#!/bin/bash

while read ifname; do 
  ofname=${ifname/.out/.cm}
  printf "IN: %s, OUT: %s\n" $ifname $ofname
  printf "running run.awk with %s & %s\n\n" $ifname $ofname

  run.awk $ifname $ofname
done < filelist.txt

This version of the script reads the input from the file, filelist.txt:

done < filelist.txt

Then for each turn of the while loop, we're using the read command to read in a line from the input file.

while read ifname; do

It then performs everything in the same way as the first script where it will run the awk script run.awk as it loops through each line of the file.

share|improve this answer
    
is there anyway i can also abstractualize this one step further, and feed this bash script the scripts from a file. I don't need it for this application but it might come in handy somewhere. Also I would delete that arrow right by run.awk right? –  user507974 Oct 1 '13 at 1:24
    
@user507974 - yes I removed the comment from the script, sorry if that was confusing you. It now only shows what you'd need. Of course you could run this using names from a file instead. I'll add that as a 2nd example. –  slm Oct 1 '13 at 1:28
    
@user507974 - see updates, let me know if you have any other questions about this answer. –  slm Oct 1 '13 at 1:39

Rather than writing a shell wrapper and spawning a new awk instance for every file you process, you can do this in awk directly. If you already have an awk script, you can access the current file using the FILENAME variable. So, if you run awk 'some commands' file1 file2, you can tell if you're working with file1 or file2 using FILENAME. You can also use > on print/printf in awk. So, if you have an awk script like

/pattern/{ print $1,$3 }

you could easily do

/pattern/{ print $1,$3 > FILENAME".processed" }

or use FNR=1 to tell when you're in a new file, and create a variable to do more complex manipulation on the filename. Like replacing a .in extension with .out, as in

sauer@humpy:/tmp$ grep . file*.in
file1.in:a
file1.in:b
file2.in:c
sauer@humpy:/tmp$ awk 'FNR=1{out=FILENAME;sub("\.in$",".out",out)} {print "processed"$0 > out}' file*.in
sauer@humpy:/tmp$ grep . file*.out
file1.out:processeda
file1.out:processedb
file2.out:processedc

I'm using grep . to show the filename and the contents of multiple files here, which is also a fun trick. But the important thing is setting the value of the out variable to a modified version of FILENAME when FNR changes to 1 (so we're on line 1 of the file), and then redirecting all prints to out. Note that this is mildly dangerous, in that failure to match the extension will result in no substitution, leading to overwriting your input files. So, it'd be good to add in a fail-safe check to make sure that out != FILENAME or something like that as well. That's left as an exercise for the reader. ;)

If you need a file containing a list of filenames, it's easiest to just run it like

awkscript $(< /path/to/filename_list_file )

Which takes the contents of the filename_list_file and puts it on the command line.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.