remove duplicate lines that are greater than

Question

So I have an input file in this format:

Hello\tWorld

I can then use

awk -F"\t" '!seen[tolower($1)]++'

to remove duplicate lines based on the first column, but how would I go about setting a condition within that statement? So i.e., only remove duplicate OR all lines if the string is used more than 5 times?

Example of input:

Hello   World
Hello   World
Hello   World
Hello   World
Hello   World
New Example
Hello   World

So in the above example column1 exists more than 5 times, so the expected output would be

Hello   World
New     Example

or if we wanted to remove that particular line entirely,

New Example

Answer 1

The expression is a boolean, testing if the array entry exists the first time. It uses ! to invert the condition. But it carries on counting.

Just adapt the test. This refuses to print the 6th and later versions.

awk -F"\t" '++seen[tolower($1)] <= 5'

Answer 2

In both examples the input file is read twice. In the first step the first column is counted and in the second step the current line is printed and compared to the count.

1. Print the first occurrence if the first column occurs more than five times:

   awk -F'\t' '
     FNR==NR{ seen[tolower($1)]++; next }
     seen[tolower($1)]
     seen[tolower($1)]>5{ delete seen[tolower($1)] }
   ' file file
   

   The current line is only printed if the first column exists in the array (seen[tolower($1)]). If it occurs more than five times, it is removed from the array.

2. Remove the line if the first column occurs more than five times:

   awk -F'\t' '
     FNR==NR{ seen[tolower($1)]++; next }
     seen[tolower($1)]<=5
   ' file file
   

   The current line is only printed if the column count is smaller or equal five.

Answer 3

Possible to do the whole thing without reading the file twice, by storing the entire file in an array, which means it will work in a pipeline. I have not compared whether the extra complexity is better than reading the file twice, but it processes abut 275,000 lines a second. I frequently use an awk array up to 400 MB, so data volume should not be an issue.

This shows the input file size, and the key counts.

Paul---) wc  53.txt
 100008  187520 1100108 53.txt
Paul---) cut -f1 53.txt | sort | uniq -c
  12500 Can
  12500 Care
  12500 If
  12500 Major
  12500 Minor
  12500 Not
      5 Oak
  12500 Sample
      1 Spruce
      2 Willow
  12500 With

This shows the execution and time. There is some debug left in to confirm the spread of data in the input, the key separation from the rest of the line, and the preservation of original sequence. Cat is used to enforce a pipelined input.

Paul---) time cat 53.txt | ./5fold
Ln   5590 Num 5 Key :Oak: Oak   Fifth
Ln   8654 Num 2 Key :Willow: Willow Pattern China
Ln  13427 Num 1 Key :Spruce: Spruce Only One
Ln  65309 Num 5 Key :Oak: Oak   Fourth
Ln  70988 Num 5 Key :Oak: Oak   Third
Ln  83982 Num 5 Key :Oak: Oak   Second
Ln  87439 Num 5 Key :Oak: Oak   First
Ln  99977 Num 2 Key :Willow: Willow Weep for Me

real    0m0.359s
user    0m0.324s
sys 0m0.048s

This the code as tested.

#! /bin/bash

AWK='''
BEGIN { FS = "\t"; nMax = 5; }
function List (Local, j) {
    for (j = 1; j in X; ++j) {
        if (N[K[j]] <= nMax)
            printf ("Ln %6d Num %d Key :%s: %s\n", j, N[K[j]], K[j], X[j]);
    }
}
{ ++N[$1]; K[NR] = $1; X[NR] = $0; }
END { List( ); }
'''
    awk -f <( echo "${AWK}" )