friends! I have a HUGE csv file and I need extract rows with headers where column called "data.status" equals "complete", then delete empty columns from it and write to another csv file. Can I achieve it with grep | sed | awk ? Thanks for answers!

EDIT 1. I create this csv from flattening about 60 000 json files and writing data from them to one csv. And I don't really know actual count of header and I can't even open it ( size of csv file about 1.5 Gb). But it has a lot of empty field and i want to delete them and make it more readable. As an example i have csv:

date.status, date.start, date.end, date.items

unsuccessful , 21-03-15, 25-03-15, some_item

complete,,,, item2

The output should looks like:

date.status, date.items

complete, item2

  • How huge is huge? How many rows? How many headers? What is the approximate file size? Either R or Python's pandas library are reasonable ways to manipulate largish CSV files for some (ill-defined) range of "large". – Faheem Mitha Mar 15 '16 at 17:14
  • Why didn't you just delete the empty columns while you were extracting from the json files? i.e. extract only the columns that you know you need? BTW, what do you want/expect to happen when different 'complete' lines have different empty fields - e.g. another line complete,21-03-15,,item3 in your input data: item2 has 2 non-empty fields, item3 has 3. – cas Mar 15 '16 at 22:34

You can do hacks with grep/sed/awk, but it is not trivial to implement a CSV parser, so I would only recommend using them for quick hacks (for example, you would have to consider all sorts of special escaping/quoting scenarios).

If you want to do this I'd suggest using a real, mature CSV parser instead, like the one provided in the Python stdlib:

import csv

with open('your.csv') as csvfile:
    reader = csv.DictReader(csvfile)
    kept_rows = (row for row in reader if row['data.status'] == 'complete')

You can then write the new rows out using a DictWriter.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.