I have several txt-files containing data, where i use grep to search for a current string of text, and use awk to filter out the variable i need. The string is repeated through the file, so i currently use this command to extract the desired string:

grep 'text' *.txt | awk ' NR==1  {print $2 } ' > outputfile

The problem is that i want to cycle through multiple files in the folder, and for each file get the extracted variable written into a single output-file. I know the question have been answered before, but I am quite fresh to this and have some difficulties implementing.

Any feedback would be highly appreciated !

  • You are selecting all files ending in .txt in that folder..as it stands it should work for if you are interested in just .txt files..If not, perhaps you should give a complete example.. – heemayl Jan 25 '16 at 7:34
  • 1
    I presume that you understand your problem perfectly, but it’s not clear to us. Can you please provide some examples of what your input files look like and what you want to get as output? You say, “I want to cycle through … files … and for each file … [write] into a single output-file.” As @heemayl says, you seem to be pretty close to that now — unless you want a single output file per input file. What other answers have you looked at; what have you tried; and how have they fallen short of the result you want?  Please do not respond in comments; edit your question to make it clearer. – G-Man Jan 25 '16 at 7:53

I'd iterate over the files in bash, keeping the file name of each, so you can redirect the output into different output files for each iteration.

For example like this (not tested):

PREFIX="/tmp/outputs"   # define where to store all the outputs
mkdir -p "${PREFIX}"    # make sure the outputs dir exists

for FILE in *.txt       # get the file names you want to work on
  # use ${PREFIX}/${FILE} to redirect output to a 
  # file that's associated with the input
  grep 'text' "${FILE}" | awk ' NR==1 {print $2 } ' > "${PREFIX}/${FILE}"
  • 2
    you don't need $(ls *.txt),using *.txt is sufficient. Also quote yours variables. – 123 Jan 25 '16 at 8:58
  • 1
    @123 thank you, and congrats for the username – replay Jan 25 '16 at 9:08

If I understand correctly, you want to do the following for each .txt file:

  • Locate the first line containing the pattern text.
  • On this line, take the second whitespace-separated field and write it out to a file whose name is related to the input file.

You aren't saying how the output file name should be constructed. I'll make it the same as the input file, but ending in .out instead of .txt.

You can do this with a shell loop.

for x in *.txt; do
  grep 'text' -- "$x" | awk '{print $2; exit}' >"${x%.*}.out"

Exiting awk as soon as it's done its job is slightly faster than telling it to keep reading but do nothing. Another possibility is to skip awk altogether and have the shell do the line splitting (whether this is faster or slower depends on so many factors that I won't hazard predictions):

for x in *.txt; do
  grep 'text' -- "$x" | read -r first second rest && printf '%s\n' "$rest" >"${x%.*}.out"

A different approach would be to do all the work in awk. Awk can act on multiple files and you can use awk's redirection for the output. This requires forking fewer processes. It's pretty straightforward in Gawk (GNU awk):

awk '/text/ {print $2 >substr(FILENAME, 1, length(FILENAME)-4) ".out"; nextfile}' *.txt

In an awk implementation that doesn't have nextfile, you need to manually handle transitions to the next file, which makes this approach less attractive (both more complex and less efficient).

awk '
    FNR==1 {first=1}
   first && /text/ {print $2 >substr(FILENAME, 1, length(FILENAME)-4) ".out"; first=0}' *.txt
  • That awk method prints from the first line of each file, not the first matching line. Without gawk you need FNR==1 {found=0} !found&&/text/ {print $2 >etc; found=1} . – dave_thompson_085 Jan 26 '16 at 7:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.