-1

I've got accumulated data in a CSV file as follows:

Parameter_1,Parameter_2,Parameter_3,Parameter_4,Parameter_5,Parameter_6
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1,Parameter_2,Parameter_3,Parameter_4,Parameter_5
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1,Parameter_2,Parameter_3
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1
Parameter_1,Parameter_2,Parameter_3,Parameter_4

How can I just read the the lines which have 4 or more parameters i.e:

Parameter_1,Parameter_2,Parameter_3,Parameter_4,Parameter_5,Parameter_6
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1,Parameter_2,Parameter_3,Parameter_4
Parameter_1,Parameter_2,Parameter_3,Parameter_4,Parameter_5
Parameter_1,Parameter_2,Parameter_3,Parameter_4
  X Parameter_1,Parameter_2,Parameter_3 X DO NOT READ THIS because it has less than 4 parameters
Parameter_1,Parameter_2,Parameter_3,Parameter_4
  X Parameter_1 X DO NOT READ THIS because it has less than 4 parameters
Parameter_1,Parameter_2,Parameter_3,Parameter_4

Ideally I would like to read lines which have more than X parameters and less than Y parameter - ie not read it if it has too many parameters in a given line

Q: How can this be written in BASH?

?A: [Should] the code look like this:

number_of_parameters=`"$line" | sed 's/[^,]//g' | wc -c` #where $line is the line thats being currently read
if [ "number_of_parameters" -gt "X" ] && [ "number_of_parameters" -lt "Y" ]
then
     ... do something
fi
1

Use grep for a much simpler solution.

grep -E '^(.*,){3}' my_csv_file.csv

Since 4 parameters or more will have at least 3 commas, hence the 3 in the regex.

(.*,) will match a parameter followed by a comma. .* means match any char (.) 0 or more times.

Here's another approach to have the values in a variable:

my_var=$(grep -E '^(.*,){3}' my_csv_file.csv)
echo "$my_var"

Make sure you have "" around $my_var otherwise you will lose the newlines.

  • Any chance you could explain what ^(.*,) does? Also how would I assign this to a variable? – 3kstc Apr 1 '15 at 3:43
  • Edited the answer with explanation – GMaster Apr 1 '15 at 3:48
  • How about assigning the value to a variable? my_var=grep -E '^(.*,){3}' my_csv_file.csv – 3kstc Apr 1 '15 at 3:50
  • Doing so will not preserve the newlines. And assigning to a variable is not recommended because if the csv file is multi gigabytes, your script might throw a memory error. Still if you want to assign to a variable another approach is required. – GMaster Apr 1 '15 at 3:57
  • I don't think the my_csv_file.csv file will exceed 100MB over the next 2 years. What would be another approach to combating this? The reason why I want a variable - is so that I can compare themy_var variable ie if the my_var is -gt 3 go ahead read the file if not try again – 3kstc Apr 1 '15 at 4:02
1

This is of course not a task that should be solved with sed, grep, or similar, but rather naturally with awk. To filter lines in a file with at least 4 (comma-separated) fields:

awk -F, 'NF>=4' my_csv_file.csv

To filter lines with, e.g., at least 4 and at most 6 (comma-separated) fields:

awk -F, 'NF>=4 && NF<=6' my_csv_file.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.