I'm trying to write a script to return duplicate values based on a set of parameters if they repeat twice. For example in my text file I have this setup:


Now if the Last Name, First Name, and DOB repeat I want it too return both those lines and continue on, I know awk can do this, but it doesn't seem to work right.

awk '!seen[$4,$5]++ > 1' DemoDATA.txt
  • show your current code – RomanPerekhrest Jun 12 '18 at 16:58
  • man sort (-u may be useful in some situations) and especially man uniq seems something you should look into. – Hannu Jun 12 '18 at 17:11
awk '!seen[$4, $5]++ > 1' DemoDATA.txt

is almost correct, apart from the fact that the fields in the file are delimited by |, and that the fields you should look at for the last name and date of birth are fields 5, 6 and 7. You also don't have to compare with 1 (I can see why you do that, but the comparison would never be true).


awk -F '|' '!seen[$5, $6, $7]++' DemoDATA.txt

This will however only output the duplicates, not the original line. To do that, you would have to save the whole file in memory as you're parsing it...

... which is what the answers to "How do I print all lines of a file with duplicate values in a certain column" are doing (or not, if they are smart, which some of them are).

| improve this answer | |

As Kusalananda suggested, saving the whole file in memory:

awk -F'|' '
        key = $5 FS $6 FS $7
        data[key] = data[key] (count[key] == 1 ? "" : ORS) $0
    END { for (key in count) if (count[key] > 1) print data[key] }
' DemoDATA.txt


perl -F'\|' -ane '
        $key = join "|", @F[4..6];
        push @{$data{$key}}, $_;
    } END {
        print @{$data{$_}} for grep {@{$data{$_}} > 1} keys %data;
' DemoDATA.txt
| improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.