0

I have an awk script named text_processing.awk . I need to pass multiple zipped files named 01JAN21.txt.gz , 02JAN21.txt.gz etc. to this script for further processing. The output file takes the date from the input file and gets named result_01JAN21.txt and so on.

Please note the files are zipped.

I tried running on one file at first using command below

zcat 01JAN21.txt.gz | awk -f text_processing.awk -

However, my output file doesn't get named, I believe because this command just opens the file and pipes it to the script, whereas, my script extracts date from the input file name as below

BEGIN{ 
  FS = ";" 
  input_file = ARGV[1] 
  sub(/\.txt\.gz/, "", input_file) 
  output = "result_" input_file ".txt 
}

Will appreciate some guidance on how to go about rectifying it. I want to keep the renaming of output file within awk script because there are multiple files I will be passing at one go.

1 Answer 1

2

Assuming you want to process all files in the current directory that have names matching *.txt.gz. Pass the name as a variable into awk and stream the uncompressed data:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -v name="$name" -f text_processing.awk
done

In the awk code, you'd use the name variable to compute the output filename.

Alternatively, get the shell script to give you the explicit output filename to use:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -v outname="result_${name%.gz}" -f text_processing.awk
done

The string result_${name%.gz} would be the name of the original file with .gz removed and result_ prepended. The awk code then uses outname to write to.

Obviously, if your awk code only writes to a single output file, you may simplify the awk code even more and have it just print to its standard output. Then do the output redirection in the shell:

for name in *.txt.gz; do
    gzip -c -d -- "$name" |
    awk -f text_processing.awk >"result_${name%.gz}"
done

Extra material requested in comments: Running the awk command in parallel.

For this, I'm going to use xargs, and I will assume that this utility supports the non-standard options -0 (for reading nul-terminated data), -r (for not executing the given command if there is no input), and -P (for running jobs in parallel).

print '%s\0' *.txt.gz |
xargs -0r -P 4 -I {} sh -c '
    gzip -c -d -- "$1" |
    awk -f text_processing.awk >"result_${1%.gz}"' sh {}

This would run the gzip+awk pipeline for four files concurrently. Add -t to the xargs utility's options to get and indication of what commands it is spawning.

4
  • Thank you, it works like a charm. Adding to this query, how can I modify your command if I want to run the files/script in parallel?
    – user511618
    Feb 3, 2022 at 7:48
  • 1
    @L1971 That's a very different question. You may want to search this site for parallel processing files. I might add something to my existing answer later when I get some free time from my day job.
    – Kusalananda
    Feb 3, 2022 at 8:00
  • 1
    @L1971 See added bit at the end.
    – Kusalananda
    Feb 3, 2022 at 11:27
  • @L1971 chameleon questions and asking multiple different questions at once are strongly discouraged. You got a good answer to the question you asked so you should accept it and ask a new question about how to run scripts in parallel if you also need help with that.
    – Ed Morton
    Feb 3, 2022 at 13:38

You must log in to answer this question.