Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have several (427 to be precise) text files with a million lines each containing 31 numbers separated be spaces (possible double spaces). However there might be some data corruption and there may be lines containing junk.

I now want to check if every line satisfies the property of containing 31 items separated by spaces (I assume that those items are numbers. A method which checks that too would be better).

My current way is

while read line;
do
   if [ $(echo "$line" | sed 's/ /\n/g' | grep -v "^$" | wc -l) -ne 31 ]
   then
      echo "$file bad";
   fi
done < $file

This replaces all spaces of a line by newlines, filters empty lines, counts the number of lines and compares it to 31.

This approach is slow and there's probably a much better way involving some fancy regular expression. What would be a faster way?

share|improve this question
    
You're gonna wait forever while million string file will be checked line by line within pure bash. –  rush Mar 27 '13 at 15:16
add comment

3 Answers

up vote 4 down vote accepted

Simply do:

awk 'NF != 31 || /[^0-9 -]/ {print FILENAME ":" FNR ": " $0}' file1 file2...

To report the lines that don't have 31 fields or have non-digits. Not as strict as @manatwork's solution as it wouldn't bark on --- or 9-8 for instance, but it might be more efficient.

share|improve this answer
    
As your regular expression checks the entire $0, better allow spaces too. –  manatwork Mar 27 '13 at 15:23
    
Oops, good point @manatwork. Edited now. –  Stephane Chazelas Mar 27 '13 at 15:56
add comment

Why not just grep alone?

bash-4.2$ cat file
1 2 -3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31
32 33 -34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 L 51 52 53 54 55 56 57 58 59 60 61 62
63 64 -65 66 67 68 69
70 71 -72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100

# listing bad lines in the file
bash-4.2$ grep -Exv '(-?[[:digit:]]+ +){30}-?[[:digit:]]+' file
32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 L 51 52 53 54 55 56 57 58 59 60 61 62
63 64 65 66 67 68 69

# listing files with bad lines
bash-4.2$ grep -Exvl '(-?[[:digit:]]+ +){30}-?[[:digit:]]+' -- *
file
share|improve this answer
    
is it possible to extend that to negative numbers as well? –  stefan Mar 27 '13 at 15:18
1  
@stefan, updated to allow negative numbers too. –  manatwork Mar 27 '13 at 15:19
    
@rush, I was thinking about that, but was not sure whether to handle it or not. Updated. Thanks. –  manatwork Mar 27 '13 at 15:21
    
can you please explain why you separated one number? –  stefan Mar 27 '13 at 15:25
    
You may need to allow leading and trailing spaces as well. –  Stephane Chazelas Mar 27 '13 at 15:28
show 1 more comment

You can read the line into an array using read -a, and then check the size of the array. This should be considerably better than spawning a subshell to fork 3 processes each line.

while read -ra line;
do
    if (( ${#line[@]} != 31 )); then
        echo "$file bad"
    fi
done < "$file"
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.