I am parsing a very large csv file, where the entry of column 26 has to be of length 10. I can see that there are cases where there are no entries (which is fine), but also cases where the entries have length less than 10 or greater than 10, that has to be erroneous. I am trying to print some of these lines to explore.

My attempt is:

awk 'length($26) < 10' my_file.csv | sort -u | cut -d ',' -f 26 | head

but this doesn't return the result I want - instead it returns a number of rows where the length of column 26 is in fact equal to 10. What am I doing wrong?

  • 5
    Shouldn't you run awk with -F, ? Jan 25, 2017 at 20:49

3 Answers 3

awk -F, 'length($26) != 10 { print }' /path/to/input > bad_field_length.txt

I would try something like:

awk -F "," 'length($26) != 10 {print $26}' my_file.csv

Then if needed you can still pipe this output through any other filters you might need like sorting etc...

Tells awk to delimit columns using commas:

-F ","

If the length of column 26 is not equal to 10, print column 26

'length($26) != 10 {print $26}' 
cat FileName|awk -F ',' 'length($3)<=9 {print $3}'|sort -u

This worked for me. Where $3 is the column I was filtering the record on, you could replace it with the column of your choice. The sort was used to sort only the unique records.


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .