6

I have a file like this:

20001 
17001
17001
53001
90001 
90001
90001

and I'm trying to modify $1 by adding one to it when it's a duplicate entry, so the output will be like this:

20001 
17001
17002
53001
90001 
90002
90003
  • 3
    Do the numbers have sufficient numerical distance that incrementing them doesn't lead to unwanted new duplicates? For example, if the original file had an additional line 90003 after your three occurences of 90001, what should happen? A related question is here, btw. – AdminBee Jul 24 at 7:51
  • no i don't have this case because that 1 in the end of each number is index – Djabri Josef Jul 24 at 13:10
10
awk '{$1+=seen[$1]++} 1' file

Add post-incremented hash value to current value of $1 before printing.

The above will produce duplicate numbers when values are close together, such as the sequence 2,2,3 – the output is 2,3,3. A loop can be used to make that 2,3,4:

awk '{while (c[$1]) {$1 += c[$1] += c[$1+c[$1]]} c[$1]++} 1'

Array c stores the offset by which $1 is to be increased (like seen in the first example). Instead of increasing $1 only by the offset for that unique value, it's also increased by the offset from the next value until a new previously unseen $1 has been reached.

| improve this answer | |
  • 3
    So much better than mine (now deleted). Or to process numbers wherever they are in the input: perl -pe 's/\d+/$& + $seen{$&}++/ge' file – Stéphane Chazelas Jul 23 at 18:07
3

A variation on @guest's answer that guards against duplicate on output by incrementing the number as long as it's already been output before:

awk '{while ($1 in c) $1 += c[$1]++; c[$1]++; print}' file

Or the same in perl, processing numbers wherever they are in the input:

perl -pe 's{\d+}{
            $i = $&;
            while (defined($c{$i})) {$i += $c{$i}++}
            $c{$i}++;
            $i
          }ge' file

On an input like:

1
1
1
5
5
10
10
1
1
1

They give:

1
2
3
5
6
10
11
4
7
8
| improve this answer | |
  • c[$1]++ : can you please explain the meaning of this and how to use it ? – Djabri Josef Jul 25 at 16:04
  • 1
    How about while (c[$1]) {$1 += c[$1] += c[$1+c[$1]] } c[$1]++? Provides a speed up in some cases (was ~32x faster to deduplicate shuf -r -i1-100 -n100000). – guest Jul 26 at 1:39
  • @guest, again, so much better than mine. If you add that to your answer, I'll delete mine (again). – Stéphane Chazelas Jul 26 at 7:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.