Extract data from csv

Question

friends! I have a HUGE csv file and I need extract rows with headers where column called "data.status" equals "complete", then delete empty columns from it and write to another csv file. Can I achieve it with grep | sed | awk ? Thanks for answers!

EDIT 1. I create this csv from flattening about 60 000 json files and writing data from them to one csv. And I don't really know actual count of header and I can't even open it ( size of csv file about 1.5 Gb). But it has a lot of empty field and i want to delete them and make it more readable. As an example i have csv:

date.status, date.start, date.end, date.items

unsuccessful , 21-03-15, 25-03-15, some_item

complete,,,, item2

The output should looks like:

date.status, date.items

complete, item2

Answer 1

You can do hacks with grep/sed/awk, but it is not trivial to implement a CSV parser, so I would only recommend using them for quick hacks (for example, you would have to consider all sorts of special escaping/quoting scenarios).

If you want to do this I'd suggest using a real, mature CSV parser instead, like the one provided in the Python stdlib:

import csv

with open('your.csv') as csvfile:
    reader = csv.DictReader(csvfile)
    kept_rows = (row for row in reader if row['data.status'] == 'complete')

You can then write the new rows out using a DictWriter.