6

I searched for this task, and found the following older questions:

But I can't use awk because my data is a complex CSV file with multiple nested double quotes.

Let's say I want to deduplicate the following (simplified case):

Ref,xxx,zzz
ref1,"foo, bar, base",qux
ref1,"foo, bar, base",bar
ref2,aaa,bbb

In the output I need it as follows:

Ref,xxx,zzz
ref1,"foo, bar, base",qux
ref2,aaa,bbb

No awk solution, please, only with any CSV parser.

I tried the following:

mlr --csv uniq -a -g Ref file.csv

But it's an error.

Improve this question

2 Answers 2

Reset to default
5

You could run

mlr --csv head -n 1 -g Ref input.csv

to get the first row grouped by Ref

Improve this answer
5

Miller's uniq sub-command returns unique records with the fields with which the uniqueness was determined. Any other fields are discarded. You are getting an error because the sub-command's -a (use all fields) and -g (use specific fields) options are incompatible.

A good solution using only Miller was already given. Using the head sub-command with -n 1 while grouping on the Ref field is probably the most convenient solution.

You may also make Miller use the same kind of operation that is commonly used in awk (!seen[$1]++), but since Miller does not have a post-increment operator, it will be slightly longer:

mlr --csv filter '@seen[$Ref] += 1; @seen[$Ref] == 1' file.csv

You could also use Miller for converting the data into JSON, then use jq to perform a unique_by() operation, and then get Miller to convert the data back into CSV:

mlr --c2j cat file.csv | jq 'unique_by(.Ref)' | mlr --j2c cat
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .