0

Essentially, what I want to do is non-destructively, run awk on every text file in a directory--and put the result with the same filename in a different directory

In other words, I want to use awk to manipulate every file in a directory, with the result being a directory containing the results of running awk in a file with the same filename.

Say I'm in directory foo, with files f1.txt, f2.txt, and f3.txt. I want to run script.awk on each of these files, with the results in bar/f1.txt, bar/f2.txt, and bar/f3.txt

2
  • 2
    What have you tried so far? What hasn't worked the way you intend, and what has worked almost-but-not-quite the way you intend?
    – John
    Aug 18, 2021 at 20:11
  • If you got an answer to your problem then please see unix.stackexchange.com/help/someone-answers for what to do next.
    – Ed Morton
    Aug 19, 2021 at 21:13

3 Answers 3

2

Not much to it, loop over the files and prefix the output directory name to the filename.

outputdir=bar
mkdir -p "$outputdir"
for f in ./*.txt; do
    awk -f script.awk < "$f" > "$outputdir/$f"
done

That assumes the input files are in the current directory, and outputdir is relative to that. So, if you have foo/ and bar/, on the same level, you need to cd foo and set outputdir=../bar.


If that's not the case, we need to remove the input directory name from $f. The prefix-removing expansion ${var#pattern} works here:

outputdir=./bar
inputdir=./foo
mkdir -p "$outputdir"
for f in "$inputdir"/*.txt; do
    awk -f script.awk < "$f" > "$outputdir/${f#"$inputdir"}"
done

Or you could use something like "$outputdir/$(basename -- "$f")" if you're more familiar with that. (Take care with the quotes if you have non-nice directory names.)

2

With GNU awk for "inplace" editing:

cp -r foo bar &&
awk -i inplace 'script' bar/*
0
0

For completeness, if you control the script you can handle this there, e.g.

 awk -vout=../bar/ '$2=="cow" {print "moo",$3,$1 > out FILENAME; next} $4!="fox" {print > out FILENAME}' *

But this will usually be a hassle and personally I would go with the other answers. As Ed notes, with other than GNU awk you probably need to separate -v and parenthesize >(out FILENAME) which are fairly minor, and change to >> and add close(out FILENAME) calls, which I consider an even worse hassle :-(

1
  • You should mention that that would require GNU awk for 3 things: a) no space between -v and out=, b) no parens around the expression that's creating the output file name, and b) not closing the output files as you go. With a POSIX awk you'll get a syntax error from the -v, a syntax error for the output file name, and once you fix those then a "too many open files" error due to not closing the output files once you cross a threshold which can be less than 20 files.
    – Ed Morton
    Aug 19, 2021 at 21:10

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .