Grep / awk on multiple files to single output

Question

I have several txt-files containing data, where i use grep to search for a current string of text, and use awk to filter out the variable i need. The string is repeated through the file, so i currently use this command to extract the desired string:

grep 'text' *.txt | awk ' NR==1  {print $2 } ' > outputfile

The problem is that i want to cycle through multiple files in the folder, and for each file get the extracted variable written into a single output-file. I know the question have been answered before, but I am quite fresh to this and have some difficulties implementing.

Any feedback would be highly appreciated !

Answer 1

I'd iterate over the files in bash, keeping the file name of each, so you can redirect the output into different output files for each iteration.

For example like this (not tested):

PREFIX="/tmp/outputs"   # define where to store all the outputs
mkdir -p "${PREFIX}"    # make sure the outputs dir exists

for FILE in *.txt       # get the file names you want to work on
do
  # use ${PREFIX}/${FILE} to redirect output to a 
  # file that's associated with the input
  grep 'text' "${FILE}" | awk ' NR==1 {print $2 } ' > "${PREFIX}/${FILE}"
done

Answer 2

If I understand correctly, you want to do the following for each .txt file:

• Locate the first line containing the pattern text.
• On this line, take the second whitespace-separated field and write it out to a file whose name is related to the input file.

You aren't saying how the output file name should be constructed. I'll make it the same as the input file, but ending in .out instead of .txt.

You can do this with a shell loop.

for x in *.txt; do
  grep 'text' -- "$x" | awk '{print $2; exit}' >"${x%.*}.out"
done

Exiting awk as soon as it's done its job is slightly faster than telling it to keep reading but do nothing. Another possibility is to skip awk altogether and have the shell do the line splitting (whether this is faster or slower depends on so many factors that I won't hazard predictions):

for x in *.txt; do
  grep 'text' -- "$x" | read -r first second rest && printf '%s\n' "$rest" >"${x%.*}.out"
done

A different approach would be to do all the work in awk. Awk can act on multiple files and you can use awk's redirection for the output. This requires forking fewer processes. It's pretty straightforward in Gawk (GNU awk):

awk '/text/ {print $2 >substr(FILENAME, 1, length(FILENAME)-4) ".out"; nextfile}' *.txt

In an awk implementation that doesn't have nextfile, you need to manually handle transitions to the next file, which makes this approach less attractive (both more complex and less efficient).

awk '
    FNR==1 {first=1}
   first && /text/ {print $2 >substr(FILENAME, 1, length(FILENAME)-4) ".out"; first=0}' *.txt