3

I am working on a shell script to move files if certain percentage of the lines contain a certain string.

I've got ~2000 files in a directory that each contain a single column of data. The number of rows in each file varies. The first row is a header, and the other rows contain numbers larger than 0, to 6 decimal places. Ex:

OMEGA
0.000010
0.000010
0.042214
0.042214
0.042214
0.042214
1.147412

I am interested in all files where at least 10% of the rows contain a value larger than 1. I am using the ratio of lines containing "1." to "." -- but I am struggling with syntax. ".omega" is the extension for the files I am interested in.

Here is what I have:

for file in *.omega
do
 if [ $(($(grep '1.' $file | wc -l) / $(grep '.' $file | wc -l)) * 100) -ge 10 ]; then
 mv $file positive_COGs/ 
 fi
done

I have played around with parenthesis/bracket placement, and have been unsuccessful. Additionally -- I am not sure if it is appropriate to use "-ge" to compare the % (not an integer) to the integer of "10" (?).

Any suggestions are much appreciated. I am using bash to execute the script.

Cheers!

  • 1
    You say "numbers larger than 0".  Are you guaranteeing that all values are less than 2?  Numbers like 3.14159 obviously don't begin with a 1. – G-Man Says 'Reinstate Monica' May 2 '15 at 4:18
  • This is not something that should be done with bash and greppatterns. Use a tool (like awk) that is suited for such tasks. – Janis May 2 '15 at 9:43
2

The problem is that bash does integer arithmetic, so if you take, say 20/50 that is always 0. So your test which does count > 1 divided by number of lines is 0, then 0 * 100 is 0, which will always be less than 10.

If you were to do the multiply by 100 before the division I think you'll get what you want.

  • 2
    +1 but it would be better to use a true numeric expression (( a >= b )) for the comparison, I think. Also if grep supports the -c flag then the wc -l could be avoided altogether e.g. if (( $(grep -c '1.' "$file") * 100 / $(grep -c '.' "$file") >= 10 )); then ... – steeldriver May 2 '15 at 1:37
  • All very good points, I agree with everything you said there. – Eric Renouf May 2 '15 at 1:38
  • @A.Lindsey glad it worked for you, if the answer solved your problem please accept it both for my benefit, but also so others will know that this question has been answered in the future--otherwise it will remain in the "unanswered" list – Eric Renouf May 2 '15 at 2:17
  • 1
    grep -c '1\.' - ('.' means any char) – Peter.O May 2 '15 at 4:02
1

With a recent version 4.x of GNU awk you could do:

awk '
  BEGINFILE { count = 0 }
  FNR == 1  { next }
  $1 > 1.0  { count++ }
  ENDFILE   { if (count/(FNR-1) >= 0.1) printf "mv %s positive_COGs\n", FILENAME }
' *.omega | sh

It initializes the counters (BEGINFILE), skips the header lines (FNR == 1), counts according to the numbers found in the data files, and prints the files (or resp. the shell commands) that match the condition (ENDFILE). The mv commands are then fed into a shell to execute the move.

0

/ in the arithmetic expression performs an integer division, i.e. it's the quotient operator. Most shells can only do integer arithmetic, not floating point arithmetic.

Instead of dividing by the total and then multiplying by 100, do the opposite. While you're at it, grep … | wc -l can be simplified to grep -c.

Furthermore, grep '1.' is wrong: it selects all lines that contain a 1 followed by another character, i.e. lines that contain a 1 somewhere other than at the very end of the line. To select lines with a number between 10 k + 1 and 10 k + 2, use grep '1\.'. To select lines containing a number that is more than 1, use grep '[1-9][0-9]*\.'.

Note that grep . selects non-empty lines. To select all lines, use wc -l. To select only lines that have a number, use something like grep '[0-9]' (this includes lines with a digit anywhere, not just line containing only a number).

if [ $(($(grep -c '[1-9][0-9]\.' <"$file") * 100 / $(grep -c '[0-9]' <"$file"))) -ge 10 ]; then

It would be potentially faster and more robust to process the file only once and count the numbers as they come. You can do that with awk.

if awk '
    $0 >= 1 {good += 1}
    $0 != 0 || $0 ~ /^ *0*\.0*$/ {total += 1}
    END {if (good < total/10) exit(1)}
'; then

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.