0

I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on the matching 1st column value.

I know awk solution to save the output to column 1 specific values.

for i in $column1values; do
  awk -F, -v a=$i '$1 == a' input.csv > output-$i.csv
done

But this solution will re-read the big csv file everytime it want to find matching column 1 value. This is inefficient considering big csv file.

So, How can I achieve this while reading the csv file only once?

EDIT:
Say my original csv file has data..

1,r1c2,r1c3,r1c4
1,r2c2,r2c3,r2c4
2,r3c2,r3c3,r3c4
2,r4c2,r4c3,r4c4
3,r5c2,r5c3,r5c4
3,r6c2,r6c3,r6c4

And I want to filter the output based on 1st column values like..

output-1.csv:

1,r1c2,r1c3,r1c4
1,r2c2,r2c3,r2c4

output-2.csv

2,r3c2,r3c3,r3c4
2,r4c2,r4c3,r4c4

output-3.csv

3,r5c2,r5c3,r5c4
3,r6c2,r6c3,r6c4

More importantly, I want to filter the output like this while reading the original big csv file only once. How to do that?

0

1 Answer 1

3

With awk:

awk -F, '{f="output-"$1".csv"; print $0 >> f; close(f)}' file
  • -F, sets the delimiter to ,.
  • f="output-"$1".csv" sets the filename to write to
  • print $0 >> f appends the the line to the file f.
  • close(f) closes the file, because there could be too many open files (thanks to comment of @Costas)
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .