5

I have a file like this:

20001 
17001
17001
53001
90001 
90001
90001

and I'm trying to modify $1 by adding one to it when it's a duplicate entry, so the output will be like this:

20001 
17001
17002
53001
90001 
90002
90003
2
  • 3
    Do the numbers have sufficient numerical distance that incrementing them doesn't lead to unwanted new duplicates? For example, if the original file had an additional line 90003 after your three occurences of 90001, what should happen? A related question is here, btw.
    – AdminBee
    Jul 24 '20 at 7:51
  • no i don't have this case because that 1 in the end of each number is index Jul 24 '20 at 13:10
10
awk '{$1+=seen[$1]++} 1' file

Add post-incremented hash value to current value of $1 before printing.

The above will produce duplicate numbers when values are close together, such as the sequence 2,2,3 – the output is 2,3,3. A loop can be used to make that 2,3,4:

awk '{while (c[$1]) {$1 += c[$1] += c[$1+c[$1]]} c[$1]++} 1'

Array c stores the offset by which $1 is to be increased (like seen in the first example). Instead of increasing $1 only by the offset for that unique value, it's also increased by the offset from the next value until a new previously unseen $1 has been reached.

1
  • 3
    So much better than mine (now deleted). Or to process numbers wherever they are in the input: perl -pe 's/\d+/$& + $seen{$&}++/ge' file Jul 23 '20 at 18:07
3

A variation on @guest's answer that guards against duplicate on output by incrementing the number as long as it's already been output before:

awk '{while ($1 in c) $1 += c[$1]++; c[$1]++; print}' file

Or the same in perl, processing numbers wherever they are in the input:

perl -pe 's{\d+}{
            $i = $&;
            while (defined($c{$i})) {$i += $c{$i}++}
            $c{$i}++;
            $i
          }ge' file

On an input like:

1
1
1
5
5
10
10
1
1
1

They give:

1
2
3
5
6
10
11
4
7
8
3
  • c[$1]++ : can you please explain the meaning of this and how to use it ? Jul 25 '20 at 16:04
  • 1
    How about while (c[$1]) {$1 += c[$1] += c[$1+c[$1]] } c[$1]++? Provides a speed up in some cases (was ~32x faster to deduplicate shuf -r -i1-100 -n100000).
    – guest
    Jul 26 '20 at 1:39
  • @guest, again, so much better than mine. If you add that to your answer, I'll delete mine (again). Jul 26 '20 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.