5

I have a file with three columns and I would like to get duplicated lines by third column, for example:

AAA = 342  
BLABLABLA = 2  
BBBx2 = 23  
1+1 = 2  
KOKOKO= 5  
2x1 = 2  

The output sould be:

BLABLABLA = 2  
1+1 = 2  
2x1 = 2  

I tried with sort and uniq, but it deletes the repeated lines and I want to print them.

5 Answers 5

10

As long as you're interested in the last column, you can do it with sort and uniq:

$ sort -k3n test.txt |  uniq  -f2 -D
1+1 = 2
2x1 = 2
BLABLABLA = 2

Here, the sort option -k3n causes the file to be sorted starting with the third field, in numeric order; the options to uniq are:

-f2   Skip the first two fields before checking for uniqueness
-D    Print all the repeated lines

Unfortunately, you cannot control the number of fields to be checked for uniqueness. You can use -w to specify a number of characters to check, but that will only help if the field you are interested in is fixed-width.

Also, beware of trailing whitespace. It will be included in the text to be checked for uniqueness.

0
0

Use Uniq.

uniq -f 3 <file name>
2
  • 1
    This is wrong. It prints the unique lines and not the duplicate lines. Commented Dec 20, 2018 at 16:21
  • Also, uniq can only work if the files is sorted.
    – AdminBee
    Commented Dec 2, 2020 at 12:41
0

@rici 's response is correct, but note that -D is a GNU extension

Here's an awk solution. It runs in a single pass, but stores the previous line:

sort -k3n <file name> | awk 'a[$3]++{ if(a[$3]==2){ print b }; print $0}; {b=$0}'
0

If your file is large, sorting and buffering may become a resource problem. For these cases, I would recommend the following two-pass approach:

awk 'NR==FNR{c[$NF]++;next} c[$NF]>1' input.txt input.txt

This will process your input file twice.

  • The first time (when NR, the global line counter, is equal to FNR, the per-file line counter), we simply increase the count of the last field on the line (array variable c, with the last field value as "array index"), and immediately skip execution.
  • The second time, we reach the c[$NF]>1 condition outside of a rule block. awk will always print the current line, including all possible modifications made so far, if it encounters a condition that evaluates to true outside of a rule block. In this case, it will print the current line if the occurence count for the last field is larger than 1 (i.e. it is a duplicate occurence).

Note that this relies on your wanting to filter by the last column. You state in your question that it should be the third column, but the line

KOKOKO= 5 

in your example input could be interpreted as having only two columns, depending on how exactly you define a "column".

0
gawk -F'= ' '{d[$2][a[$2]++]=$0} END{for (i in a) {if (a[i] > 1) for (j in d[i]) {print d[i][j]}}}'

Explanation:

For each line ($0) take the second field ($2) separated by '= ' and use this field as a key to a hash 'a' to count occurrences of this field, also use this field as the first dimension key for a two-dimensional hash 'd', and the value of hash 'a' referenced by this field as the second dimension key to store the value of the current line ($). At the end loop through all elements of 'a' with values grater than one (means duplicate) and print corresponding values of 'd'.

NOTE: This one liner does not require sorting, but consumes memory.

1
  • 1
    Would you like to provide a sentence or two explaining how this works? You're more likely to get some points for that Commented Dec 15, 2020 at 22:54

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .