Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am using grep at the end of a set of piped commands to filter some rows out of a file. Afterwards I will want to do some arithmetic based on the number of remaining rows.

e.g.

chrN="chr1|chr2|chr3|chr4|chr5|chr6|chr7"
otherCommands | grep -Ew $chrN  > $ChIP".bed"
count_pos=`wc -l $ChIP"bed" | awk '{print 1000000/$1}'`

It strikes me as very wasteful after piping together lots of command to then write it to a file and then read through it again just to collect one number. .. but I don't know how to do that.

So my question is how do I save the number of rows that passed the grep filter to the variable count_pos and save the filtered rows to the file too?

share|improve this question
add comment

2 Answers 2

up vote 5 down vote accepted

You can use a tee to write to both a file and stdout:

chrN="chr1|chr2|chr3|chr4|chr5|chr6|chr7"
count_pos=$(otherCommands |
  grep -Ew "$chrN" |
  tee "$ChIP.bed" |
  wc -l |
  awk '{print 1000000/$1}')
share|improve this answer
    
OK, thats a great answer.. if I could just push it a little further?.. say I wanted not to write to the file but instead save the count in count_pos then use that as a variable in the next command of my now very long pipe? –  Stephen Henderson Mar 14 at 16:41
    
@Stephen, Not sure what you mean. This will save the count in count_pos ($() is usually a better option than backticks). –  Graeme Mar 14 at 16:46
    
No forget what I said.. I have to pass through all the data to count it .. Doh! So I might as well stop the pipe there. Apols and thx again.. –  Stephen Henderson Mar 14 at 16:49
add comment

Do everything with awk. Initialize counter, count and print (grep+count) with pattern matching and output count at the end:

awk 'BEGIN { c=0} pattern{ c++; print $0 } END{ print 1000000/c }'

You don't have to print if you don't want to. Or, you could use print $0 > output_file inside awk to print grepped results in a file, and count on stdout.

share|improve this answer
    
thx, that is a good option but I think the above is exactly what I was looking for. –  Stephen Henderson Mar 14 at 16:37
2  
This replaces grep, tee, wc and awk in a single command. –  orion Mar 14 at 16:40
    
indeed you maybe right, it maybe better, except for my innate fear of awk ;) I will test it though! –  Stephen Henderson Mar 14 at 16:43
    
Then why use awk to do arithmetics? :) I used to fear awk too. But it's just a very logical and powerful tool - it's like perl, except it's easier to read and works better in a pipe. –  orion Mar 14 at 16:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.