3

I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates.

For example, a file like this where I would like to remove the duplicates of AAA

AAA
BBB
AAA
BBB
CCC

should become

AAA
BBB
BBB
CCC

I guess I should use sed but I have no idea how to write the command.

4
  • 1
    BBB is also duplicated, but you want a solution that takes AAA as an argument?
    – Kusalananda
    Aug 25, 2017 at 15:42
  • Is the first occurrence of AAA always on the first line? Aug 25, 2017 at 15:56
  • I know I've answered this several times. Here's one: vi.stackexchange.com/a/6248/4676
    – Wildcard
    Aug 25, 2017 at 23:58
  • AAA would be an argument
    – neXus
    Aug 28, 2017 at 8:08

3 Answers 3

7

With GNU sed:

sed '0,/^AAA$/b;//d'

That is, let everything through (b branches off like a continue) up to the first AAA (from the 0th line (that is even before the first line) and the first one matching /^AAA$/ (which could be the first line)), and then for the remaining lines, delete every occurrence of AAA (an empty // pattern reuses the last pattern).

GNU sed is needed for the 0 address (and the ability to have other commands after the b one in the same expression, though that could be easily worked around in other implementations by using two -e expressions)

With awk:

awk '$0 != "AAA" || !n++'

(or for a regexp pattern: awk '!/^AAA$/ || !n++')

a shorthand for:

awk '! (&0 == "AAA" && count > 0) {print; count++}'
5
  • I like that awk shorthand. Took a second to parse this early in the day, but I've filed that away for future use.
    – DopeGhoti
    Aug 25, 2017 at 15:59
  • +1 for the genius sed solution
    – Philippos
    Aug 25, 2017 at 16:32
  • I'm trying to wrap my head around your sed solution. It does work, but how does it know how to delete only the AAA line?
    – neXus
    Aug 28, 2017 at 11:11
  • @neXus, see if the new edit makes it clearer. Aug 28, 2017 at 11:14
  • Yes, it does. Thank you. I didn't know that the empty //d pattern reused the previous one. I assumed it deleted everything. This makes much more sense. Beautiful solution!
    – neXus
    Aug 28, 2017 at 11:18
2

Stéphane Chazelas' awk solution is beautiful:

awk '!/AAA/ || !n++' file.in

This may be generalized as

awk '$0 !~ pattern || !n++' pattern="$pattern" file.in

for a given shell variable $pattern containing some regular expression.

If $pattern contains backslashes, these need to be escaped (\\), or you may use

P="$pattern" awk '$0 !~ ENVIRON["P"] || !n++' file.in
0
0

Just exchange buffers each time you encounter that line, if the pattern space contains the same line delete it, else retrieve the line from the hold buffer:

sed -e '/^AAA$/{x;//d;g' -e'}' infile

or

sed '/^AAA$/{
x
//d
g
}' infile

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .