0

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on the matching 1st column value.

I know awk solution to save the output to column 1 specific values.

for i in $column1values; do
  awk -F, -v a=$i '$1 == a' input.csv > output-$i.csv
done

But this solution will re-read the big csv file everytime it want to find matching column 1 value. This is inefficient considering big csv file.

So, How can I achieve this while reading the csv file only once?

EDIT:
Say my original csv file has data..

1,r1c2,r1c3,r1c4
1,r2c2,r2c3,r2c4
2,r3c2,r3c3,r3c4
2,r4c2,r4c3,r4c4
3,r5c2,r5c3,r5c4
3,r6c2,r6c3,r6c4

And I want to filter the output based on 1st column values like..

output-1.csv:

1,r1c2,r1c3,r1c4
1,r2c2,r2c3,r2c4

output-2.csv

2,r3c2,r3c3,r3c4
2,r4c2,r4c3,r4c4

output-3.csv

3,r5c2,r5c3,r5c4
3,r6c2,r6c3,r6c4

More importantly, I want to filter the output like this while reading the original big csv file only once. How to do that?

3

With awk:

awk -F, '{f="output-"$1".csv"; print $0 >> f; close(f)}' file
  • -F, sets the delimiter to ,.
  • f="output-"$1".csv" sets the filename to write to
  • print $0 >> f appends the the line to the file f.
  • close(f) closes the file, because there could be too many open files (thanks to comment of @Costas)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.