2

I am using grep at the end of a set of piped commands to filter some rows out of a file. Afterwards I will want to do some arithmetic based on the number of remaining rows.

e.g.

chrN="chr1|chr2|chr3|chr4|chr5|chr6|chr7"
otherCommands | grep -Ew $chrN  > $ChIP".bed"
count_pos=`wc -l $ChIP"bed" | awk '{print 1000000/$1}'`

It strikes me as very wasteful after piping together lots of command to then write it to a file and then read through it again just to collect one number. .. but I don't know how to do that.

So my question is how do I save the number of rows that passed the grep filter to the variable count_pos and save the filtered rows to the file too?

5

You can use a tee to write to both a file and stdout:

chrN="chr1|chr2|chr3|chr4|chr5|chr6|chr7"
count_pos=$(otherCommands |
  grep -Ew "$chrN" |
  tee "$ChIP.bed" |
  wc -l |
  awk '{print 1000000/$1}')
  • OK, thats a great answer.. if I could just push it a little further?.. say I wanted not to write to the file but instead save the count in count_pos then use that as a variable in the next command of my now very long pipe? – Stephen Henderson Mar 14 '14 at 16:41
  • @Stephen, Not sure what you mean. This will save the count in count_pos ($() is usually a better option than backticks). – Graeme Mar 14 '14 at 16:46
  • No forget what I said.. I have to pass through all the data to count it .. Doh! So I might as well stop the pipe there. Apols and thx again.. – Stephen Henderson Mar 14 '14 at 16:49
6

Do everything with awk. Initialize counter, count and print (grep+count) with pattern matching and output count at the end:

awk 'BEGIN { c=0} pattern{ c++; print $0 } END{ print 1000000/c }'

You don't have to print if you don't want to. Or, you could use print $0 > output_file inside awk to print grepped results in a file, and count on stdout.

  • thx, that is a good option but I think the above is exactly what I was looking for. – Stephen Henderson Mar 14 '14 at 16:37
  • 2
    This replaces grep, tee, wc and awk in a single command. – orion Mar 14 '14 at 16:40
  • indeed you maybe right, it maybe better, except for my innate fear of awk ;) I will test it though! – Stephen Henderson Mar 14 '14 at 16:43
  • Then why use awk to do arithmetics? :) I used to fear awk too. But it's just a very logical and powerful tool - it's like perl, except it's easier to read and works better in a pipe. – orion Mar 14 '14 at 16:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.