How to delete partial line matches in a file?

Question

I want to walk a file and compare two lines to see if they begin with the same 12 characters. If they do, I want to delete the first line and then compare the remaining line with the next line in the file until all lines have been compared.

The file contains the list of files in the directory, already sorted. There can be two or more files (always in sequence) that start with the same 12 characters. I only want the last one.

I saw a similar solution, in an early post:

sed '$!N; /\(.*\)\n\1:FOO/D; P;D' file

but I could not modify it to work for me.

Answer 1

If your system has uniq, you can use the -w n option to limit the comparison to the first n characters of each line:

uniq -w 12 file.txt

That will return the first consecutive occurrence of each match.

If you want the last consecutive occurrence of each match, run the file through tac to reverse the order of lines, before and after processing it with uniq:

tac file.txt | uniq -w 12 | tac

Examples:

[fish/]$ cat fish.txt
Apopterygion_alta.fasta
Apopterygion_oculus.fasta
Oncorhynchus_gorbuscha.fasta
Oncorhynchus_keta.fasta
Oncorhynchus_kisutch.fasta
Oncorhynchus_masou.fasta
Oncorhynchus_nerka.fasta
Oncorhynchus_tshawytscha.fasta

[fish/]$ uniq -w 12 fish.txt
Apopterygion_alta.fasta
Oncorhynchus_gorbuscha.fasta

[fish/]$ tac fish.txt | uniq -w 12 | tac
Apopterygion_oculus.fasta
Oncorhynchus_tshawytscha.fasta

Answer 2

That sed, adapted:

sed '$!N; /^\(.\{12\}\).*\n\1/D; P;D'

On any lines but the last ($!), it loads the next line (N), then looks for the repeating 12 characters in the pattern buffer (at start of line ^, capture \(\) 12 characters .\{12\}, then match anything .*, then a newline \n, and the captured group again \1), and if found, deletes the first line in the buffer (D); Then it prints (P) and deletes (D) the (remaining) first line in the buffer.

$ cat test
foo
barbarbarbar123
barbarbarbar456
doo
$ sed '$!N; /^\(.\{12\}\).*\n\1/D; P;D'  < test
foo
barbarbarbar456
doo

---

Rather straightforward in awk too:

$ awk 'NR != 1 && substr($0, 0, 12) != substr(p, 0, 12) {print p} 
     {p=$0} END {print p}' < test
foo
barbarbarbar456
doo

On all but the first line (NR != 1), compare the first 12 characters of the current line $0 and the string p, if they're different, print p. Then set p to the current line. In the END, print p in any case.