Possible to do the whole thing without reading the file twice, by storing the entire file in an array, which means it will work in a pipeline. I have not compared whether the extra complexity is better than reading the file twice, but it processes abut 275,000 lines a second. I frequently use an awk array up to 400 MB, so data volume should not be an issue.
This shows the input file size, and the key counts.
Paul---) wc 53.txt
100008 187520 1100108 53.txt
Paul---) cut -f1 53.txt | sort | uniq -c
12500 Can
12500 Care
12500 If
12500 Major
12500 Minor
12500 Not
5 Oak
12500 Sample
1 Spruce
2 Willow
12500 With
This shows the execution and time. There is some debug left in to confirm the spread of data in the input, the key separation from the rest of the line, and the preservation of original sequence. Cat is used to enforce a pipelined input.
Paul---) time cat 53.txt | ./5fold
Ln 5590 Num 5 Key :Oak: Oak Fifth
Ln 8654 Num 2 Key :Willow: Willow Pattern China
Ln 13427 Num 1 Key :Spruce: Spruce Only One
Ln 65309 Num 5 Key :Oak: Oak Fourth
Ln 70988 Num 5 Key :Oak: Oak Third
Ln 83982 Num 5 Key :Oak: Oak Second
Ln 87439 Num 5 Key :Oak: Oak First
Ln 99977 Num 2 Key :Willow: Willow Weep for Me
real 0m0.359s
user 0m0.324s
sys 0m0.048s
This the code as tested.
#! /bin/bash
AWK='''
BEGIN { FS = "\t"; nMax = 5; }
function List (Local, j) {
for (j = 1; j in X; ++j) {
if (N[K[j]] <= nMax)
printf ("Ln %6d Num %d Key :%s: %s\n", j, N[K[j]], K[j], X[j]);
}
}
{ ++N[$1]; K[NR] = $1; X[NR] = $0; }
END { List( ); }
'''
awk -f <( echo "${AWK}" )
Hello
on the 4th line and the 5thHello
on the 10th line?