After running csvcut on a comma-delimited .csv file (downloadable here):

[root@server files]# csvcut -c title,mpn,overview,techspecs2,image_carousel_elargesrc syn_multi-image.csv > syn_scraped_cut.csv

I get the error:

CSV contains fields longer than maximum length of 131072 characters. Try raising the maximum with the field_size_limit parameter, or try setting quoting=csv.QUOTE_NONE.

Though large, I can tell you for sure that my longest field is only 65535 characters long, which is under the maximum allowed length by a pretty safe margin.

I have no idea what setting quoting=csv.QUOTE_NONE refers to. I have only been using simple csvkit commands and that is all I know.

Reading similar threads and answers such as here and here, I am unable to extract any kind of solution in the context of csvkit, specifically. I'm not adept at programming in general and am limited to using csvkit, its commands and options.

How do I fix this error?

  • Probably there is some error in the CSV. Could you share a portion of it and before sharing it check that you have the same error applying csvcut on it?
    – aborruso
    Jul 21 '20 at 6:44

The documentation of the csvkit states that the -z argument can be used to adjust the FIELD_SIZE_LIMIT.

So I suggest trying with a fairly high number:

csvcut -z 2500000 -c title,mpn,overview,techspecs2,image_carousel_elargesrc syn_multi-image.csv > syn_scraped_cut.csv

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.