I'm working with some CSV files generated by YouTube (so I cannot change the source structure). In the CSV file, some records span multiple lines. A hypothetical example with many other columns omitted for brevity is as follows:
video_id, upload_time, title, policy oHg5SJYRHA0, 2007/05/15, "RickRoll'D", "Monetize in all countries except: CU, IR, KP, SD, SY Track in countries: CU, IR, KP Block in countries: SD, SY" dQw4w9WgXcQ, 2009/10/24, "Rick Astley - Never Gonna Give You Up", "Monetize in all countries except: CU, IR, KP, SD, SY Track in countries: CU, IR, KP, SD, SY"
A typical file contains hundreds of thousands of records if not millions of records (one file is 29.57GB in size), which is too big to process in one go, so I would like to split them up into smaller chunks for processing on separate machines. I've previously used
-l on other report files and that works great when there is no newline in cells. In this case, if the split happens on a bad line (e.g.: line 4 of the example), then I have broken records in two files. Short of parsing the CSV file and then rebuilding it into multiple files, is there an effective way to split CSVs like this?