Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'd like to remove from a given column ($2 in the example) the duplicate fields (comma separated), but only if the ID column ($1 in the example) is the same.


A    1,2,3,4
A    8,9,10,11
A    2,3,4,11,12
B    4,5,6,7
B    6,8,9,10

Expected output:

A    1,2,3,4
A    8,9,10,11
A    12
B    4,5,6,7
B    8,9,10
share|improve this question
up vote 7 down vote accepted

This is easy enough in awk using an array, split, and a regular loop:

    split($2, elements, ",")
    out = ""
    for (i in elements) {
        el = elements[i]
        key = $1 " " el
        if (!(key in used)) {
            out = out el ","
        used[key] = 1
    sub(/,$/, "", out)
    $2 = out

For each line, we split the second column by commas and save the bits into an array elements. We then build up the new value for that column with a loop, checking whether we've seen the value before or not.

We're keeping the set of values we've already seen in the (associative) array used. We construct a string key containing both the value of the first column and the value from the second we're currently looking at. If key is in used, we've seen this one before for this ID and shouldn't put it in the output; otherwise, it's new, and we concatenate the value to out. So that we don't use it again, we store key (which will be something like "A 3") into our set of seen elements.

Finally, we put the constructed list back into the second column. This is essentially the approach you'd take in any other language.

Put the code above into a file and run it with awk -f, or single-quote all of it as an argument on the command line.

share|improve this answer
You may want to use a space instead of _ as that's guaranteed not to be found in the fields. Note that it doesn't preserve the spacing of the fields. – Stéphane Chazelas Aug 21 '14 at 9:55
perl -lpe 's/(\S+)\s+\K\S+/join ",", grep {!$seen{$1}->{$_}++} split ",", $&/e'
share|improve this answer

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.