1

Basically i do have a large text file on linux server and i would like to count duplicate for entries based on first 6 digits only:

Input:

1111110000000222982
112114
1111119292828
11101110
112114

Output:

111111 (2)
112114 (2)

what is the best solution to handle that case?

3
  • @RomanPerekhrest take a look if you available. Commented Nov 16, 2017 at 14:34
  • cat file | awk '{ print substr($0, 0, 6)}' |sort |uniq -c Commented Nov 16, 2017 at 14:43
  • or ... grep -Eo '^.{6}' file | sort | uniq -cd
    – JJoao
    Commented Nov 16, 2017 at 14:45

2 Answers 2

2

With some uniq implementations, you can do:

$ <file cut -c-6 | sort | uniq -cd
   2 111111
   2 112114

Not all implementations support combining -c (count) with -d (only output duplicates).

On those that don't, you can use uniq -c and pipe to awk '$1 > 1', or do the whole thing in awk which will save you the sorting (but at the potential cost of using a lot of memory if there are a lot of unique strings (though here it's only up to 1 million different 6 character strings which is not a lot))

0
0
awk '
 {a[substr($0,1,6)]++} 
 END {for (i in a) {if (a[i] > 1) printf "%s (%d)\n", i, a[i]}}
' file
111111 (2)
112114 (2)
0

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .