So I have a text file that I need to get extract specific lines out of and count the number of times a number in a specific column appears. I have about 100 or so of these files. I can get it done in small steps but want to get it done using bash/ksh:

foreach i *h3
sed '4p;55p;77q;d' $i >> output.txt

^^^^this will just extract the lines I need from each h3 file

awk '{print $6}' output.txt | grep 'P2' | wc -l

^^^this will just extract column 6 from output.txt and count the number of times P2 appears in column 6

Is there a way I can combine all of this into a bash/ksh script?


If I understood correctly:

  • you want to count how many times there is "P2" anywhere within the 6th field of lines 4,55 and 77 of a few files (named *h3)?

You could do this with 1 awk:

awk '
( FNR==4 || FNR==55 || FNR==77 ) {
    if ( $6 ~ "P2" ) { occurence++ } 
    printf "There was: %d P2 ", occurence
    printf " among the 6th field on lines 4,55 or 77 of the *h3 files\n"
}' *h3

Note: change $6 ~ "P2" into $6 == "P2" if you want an exact match (instead of a grep, as you did use in your own example, so that it also matches: somethingP2otherthing and variants thereof)

FNR = File's Number of Records = number of lines into the current file (ie, starts again at 1 at each file's first line) (Current file whose name can also be known by the internal variable: FILENAME)

(NR = here would not work, as it is the (total) Number or Records read since the beginning (not since the beginning of the current file) )

  • 2
    +1. can be shortened using a regexp match for FNR: FNR ~ /^(4|55|77)$/ && $6 ~ /P2/ {occurence++}; END{ .... } – cas May 20 '16 at 2:43

Sure. Here's one way

for f in *h3; do
    for ((n=1; n<=77; n++)); do
        IFS= read -r line
        if [[ $n == 4|55|77 ]]; then
            echo "$line"
            set -f
            set -- $line
            set +f
            if [[ $6 == *P2* ]]; then
    done < "$f"
done > output.txt
echo "saw P2 in 6th column $p2_count times"
  • Kudos for a pure shell version – Henk Langeveld May 22 '16 at 11:27

Or using a bash one-liner:

for i in *h3; do sed '4p;55p;77q;d' $i | awk '{print $6}' | grep 'P2'; done | wc -l

Or shorter using grep -c:

for i in *h3; do sed '4p;55p;77q;d' $i | awk '{print $6}'; done | grep -c 'P2'

Usually, when a question asks "how do I process a bunch of text files using specific tool(s) in a bash loop?", the answer is, in part, "Don't use a bash loop, use (some or all of) the tool(s) themselves". Sometimes part of the answer is even "Don't use those tools, use this instead".

What you want can be done with awk alone, no need for a shell loop. Or sed or grep or wc:

awk 'BEGIN {OFS="\t"}
     FNR ~ /^(4|10|17)$/ && $6 ~ /P2/ {count++}
     ENDFILE { print FILENAME, count; count=0 }' *h3

Note: ENDFILE is specific to GNU awk. It won't work with other versions of awk.

And this version also prints a cumulative total for all files:

awk 'BEGIN {OFS="\t"}
     FNR ~ /^(4|10|17)$/ && $6 ~ /P2/ {count++; total++}
     ENDFILE { print FILENAME, count; count=0 }
     END { print "---", total,"total" }' *h3

The END{} block prints the total and also makes a crude attempt to distinguish the actual total from any files that happen to have the filename "total". It does this by printing --- in the first field, then the total, and then the string total in the third field. This is far from perfect but is good enough in many cases. It's better than, like wc, not making the attempt at all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.