14

I've got a huge (about half a GiB, impossible to use a usual text editor on) CSV file with fields enclosed in double quotes like "abc","def" but need a file without quotes (I am sure this is not going to break the file consistency - a comma is never used inside the values in it).

How to remove all the quotes (without introducing spaces on their places)?

3 Answers 3

22

tr can do that:

tr -d \" < infile > outfile

You could also use sed:

sed 's/"//g' < infile > outfile
5
  • 1
    Why did you removed < infile > outfile? IMHO it was more informative.
    – Ivan
    Dec 14, 2012 at 9:27
  • @Ivan I thought it was probably implicit on second thought.
    – Chris Down
    Dec 14, 2012 at 9:29
  • Only when for experienced command line users. Even though I a have used < before a couple of times (to import SQL scripts into MySQL and SQLite) It would be immediately obvious for me that I should use < in this case. I think it would be better to return the full command line example back for further reference of those who may need it.
    – Ivan
    Dec 14, 2012 at 9:35
  • @Ivan and @Chris, I've reverted it back (pending review) to include < infile > outfile, hope that's OK. Dec 14, 2012 at 11:22
  • Thanks, @donothingsuccessfully. I've totally mistyped the previous comment, that's a pity I can't edit it %-]
    – Ivan
    Dec 14, 2012 at 13:14
7

Another version of sed command:

sed -i s/\"//g file.txt
  • sed stream editor

    • -i in-place (edit file in place)
    • s the substitute command
    • /replacement_from_reg_exp/replacement_to_text/ statement
    • \" quotes preceded by backslash (replacement_from_reg_exp)
    • empty string between slash delemiters (replacement_to_text)
    • g global (for replace all occurrence in line)
  • file.txt the file name

0

You don't need to remove the double quotes if you read the data using a CSV-aware library. The rest of this answer shows how you may remove the unneeded double quotes from a generic CSV file, for the case where you rely on a more simplistic approach to parsing the data later.

Miller (mlr) would be able to read the CSV data and output it again using just

mlr --csv cat file

(Use with -I to do "in-place" editing.)

Miller would by default only quote fields that require quoting, e.g. fields containing embedded commas, quotes, or newlines.

In the same fashion, csvformat from csvkit would similarly only by default quote fields that actually require quoting:

csvformat file

The csvformat utility can't do "in-place" editing, so you would need to redirect the output to a new name.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .