1

So I have an input file in this format:

Hello\tWorld

I can then use

awk -F"\t" '!seen[tolower($1)]++'

to remove duplicate lines based on the first column, but how would I go about setting a condition within that statement? So i.e., only remove duplicate OR all lines if the string is used more than 5 times?

Example of input:

Hello   World
Hello   World
Hello   World
Hello   World
Hello   World
New Example
Hello   World

So in the above example column1 exists more than 5 times, so the expected output would be

Hello   World
New     Example

or if we wanted to remove that particular line entirely,

New Example
  • Your command already gives your first expected output. How do you want to get the final output you show? Are you trying to remove any lines whose first field is present more than once? – terdon Dec 4 '19 at 9:27
  • Yeah but I only want to remove the duplicates if the column exists more than 5 times. – Bash Dec 4 '19 at 9:29
  • So atm it will do it for any column that is present more than once, but I only want to do it if the column is present more than 5 times – Bash Dec 4 '19 at 9:30
  • And will the file always be sorted or can you have the 4th Hello on the 4th line and the 5th Hello on the 10th line? – terdon Dec 4 '19 at 9:31
  • 5th of the 10th, it's random – Bash Dec 4 '19 at 9:38
0

The expression is a boolean, testing if the array entry exists the first time. It uses ! to invert the condition. But it carries on counting.

Just adapt the test. This refuses to print the 6th and later versions.

awk -F"\t" '++seen[tolower($1)] <= 5'
| improve this answer | |
  • Thank you for the help :)! – Bash Dec 4 '19 at 10:27
  • I didn't see the option to remove all copies if there are 6 or more. That does not work with this technique, because those first 5 lines have already gone to output. So you need to see all the input to decode: You might have "Hello" on the first 5 lines and the millionth one too. Two ways to fix (a) Store all the complete lines in an array and decide which to output later. (b) Read the input twice, count the first time and test on the second. Depends on data volume, and whether it has to work in a pipeline. – Paul_Pedant Dec 4 '19 at 10:37
0

In both examples the input file is read twice. In the first step the first column is counted and in the second step the current line is printed and compared to the count.

  1. Print the first occurrence if the first column occurs more than five times:

    awk -F'\t' '
      FNR==NR{ seen[tolower($1)]++; next }
      seen[tolower($1)]
      seen[tolower($1)]>5{ delete seen[tolower($1)] }
    ' file file
    

    The current line is only printed if the first column exists in the array (seen[tolower($1)]). If it occurs more than five times, it is removed from the array.

  2. Remove the line if the first column occurs more than five times:

    awk -F'\t' '
      FNR==NR{ seen[tolower($1)]++; next }
      seen[tolower($1)]<=5
    ' file file
    

    The current line is only printed if the column count is smaller or equal five.

| improve this answer | |
0

Possible to do the whole thing without reading the file twice, by storing the entire file in an array, which means it will work in a pipeline. I have not compared whether the extra complexity is better than reading the file twice, but it processes abut 275,000 lines a second. I frequently use an awk array up to 400 MB, so data volume should not be an issue.

This shows the input file size, and the key counts.

Paul---) wc  53.txt
 100008  187520 1100108 53.txt
Paul---) cut -f1 53.txt | sort | uniq -c
  12500 Can
  12500 Care
  12500 If
  12500 Major
  12500 Minor
  12500 Not
      5 Oak
  12500 Sample
      1 Spruce
      2 Willow
  12500 With

This shows the execution and time. There is some debug left in to confirm the spread of data in the input, the key separation from the rest of the line, and the preservation of original sequence. Cat is used to enforce a pipelined input.

Paul---) time cat 53.txt | ./5fold
Ln   5590 Num 5 Key :Oak: Oak   Fifth
Ln   8654 Num 2 Key :Willow: Willow Pattern China
Ln  13427 Num 1 Key :Spruce: Spruce Only One
Ln  65309 Num 5 Key :Oak: Oak   Fourth
Ln  70988 Num 5 Key :Oak: Oak   Third
Ln  83982 Num 5 Key :Oak: Oak   Second
Ln  87439 Num 5 Key :Oak: Oak   First
Ln  99977 Num 2 Key :Willow: Willow Weep for Me

real    0m0.359s
user    0m0.324s
sys 0m0.048s

This the code as tested.

#! /bin/bash

AWK='''
BEGIN { FS = "\t"; nMax = 5; }
function List (Local, j) {
    for (j = 1; j in X; ++j) {
        if (N[K[j]] <= nMax)
            printf ("Ln %6d Num %d Key :%s: %s\n", j, N[K[j]], K[j], X[j]);
    }
}
{ ++N[$1]; K[NR] = $1; X[NR] = $0; }
END { List( ); }
'''
    awk -f <( echo "${AWK}" ) 
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.