I have a gigantic csv file that is around a GB in size. The csv's 1st column contains around 300 unique integer values. I want to process the csv once and save the output to different files based on the matching 1st column value.
awk solution to save the output to column 1 specific values.
for i in $column1values; do awk -F, -v a=$i '$1 == a' input.csv > output-$i.csv done
But this solution will re-read the big csv file everytime it want to find matching column 1 value. This is inefficient considering big csv file.
So, How can I achieve this while reading the csv file only once?
Say my original csv file has data..
1,r1c2,r1c3,r1c4 1,r2c2,r2c3,r2c4 2,r3c2,r3c3,r3c4 2,r4c2,r4c3,r4c4 3,r5c2,r5c3,r5c4 3,r6c2,r6c3,r6c4
And I want to filter the output based on 1st column values like..
More importantly, I want to filter the output like this while reading the original big csv file only once. How to do that?