I'm looking for code to removing duplicate lines after comparing the first tab-delimited pattern:

Example : ( = tab)

car ↦ ab ↦ 38/43
car ↦ ca ↦ 2/34
house ↦ ca ↦ 34/34
house ↦ d ↦ 2/3
house ↦ ht ↦ 3/7
boat ↦ as ↦ 5/5

I want to delete to duplicate lines, with the constraint that the line with ca in the second field should not be deleted.


car ↦ ca ↦ 2/34
house ↦ ca ↦ 34/34
boat ↦ as ↦ 5/5

Any ideas? I tried sed and sort, but I don't get any working code.

  • Are all the car entries grouped together, or could you have the lines car ab …, house ca …, car ca … and you'd need to delete the first line? – Gilles Nov 18 '14 at 7:33

I assume that if there are duplicate entries there will always be one with ca as the second field.

In your example data all the lines that have the same first field are grouped together, but you didn't mention whether that's always the case. If it is, then the task is a little simpler, but the awk script below will work even if the matching lines aren't grouped together.


#!/usr/bin/awk -f

    if (!($1 in lines) || ($2 == "ca"))
        lines[$1] = $0

    for (i in lines) 
        print lines[i]

If the input file is called data, you'd run it like:

awk -f dedup.awk data


awk -F '\t' -f dedup.awk data

to ensure that it uses tab as the field separator; the default field separator is one or more spaces &/or tabs. See Fields in the gawk man page for further info.

Due to the way that awk arrays work, the output data will probably not be in the same order as the source data. The original order could be preserved with a slightly more complicated program, or the output could be sorted using one of awk's sort functions, but it's probably more flexible to use the bash sort command.

That awk script can be condensed, if you want:

awk '{if(!($1 in a)||($2=="ca"))a[$1]=$0};END{for(i in a)print a[i]}' data
  • If you'd like to condense a little bit more cut the if from script awk '!($1 in a)||$2=="ca"{a[$1]=$0}... – Costas Nov 17 '14 at 9:58
  • @Costas: Good point, although you've lost some parentheses, so your version doesn't work properly. We need: '!($1 in a)||($2=="ca"){a[$1]=$0};END{for(i in a)print a[i]}' – PM 2Ring Nov 17 '14 at 10:06
  • Yes, for portability reason you are right, but in my bash it works without parentheses added by you and without ; before END. But I use GNU version - mawk – Costas Nov 17 '14 at 10:24
  • @Costas: My apologies! I just noticed that when I tried your version I somehow put spaces around the ==. And that only printed `boat as 5/5'. But if get rid of those spaces I do get the proper output. FWIW, I'm using gawk: GNU Awk 3.1.7. But I think I prefer the extra parentheses as it makes the precedence clear. – PM 2Ring Nov 17 '14 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.