I'm looking for a way to remove one specific line from a bunch of files, but only if it occurs more than once in that file. Other lines should be kept, even if they are duplicates.

For example, a file like this where I would like to remove the duplicates of AAA


should become


I guess I should use sed but I have no idea how to write the command.

  • 1
    BBB is also duplicated, but you want a solution that takes AAA as an argument?
    – Kusalananda
    Aug 25 '17 at 15:42
  • Is the first occurrence of AAA always on the first line? Aug 25 '17 at 15:56
  • I know I've answered this several times. Here's one: vi.stackexchange.com/a/6248/4676
    – Wildcard
    Aug 25 '17 at 23:58
  • AAA would be an argument
    – neXus
    Aug 28 '17 at 8:08

With GNU sed:

sed '0,/^AAA$/b;//d'

That is, let everything through (b branches off like a continue) up to the first AAA (from the 0th line (that is even before the first line) and the first one matching /^AAA$/ (which could be the first line)), and then for the remaining lines, delete every occurrence of AAA (an empty // pattern reuses the last pattern).

GNU sed is needed for the 0 address (and the ability to have other commands after the b one in the same expression, though that could be easily worked around in other implementations by using two -e expressions)

With awk:

awk '$0 != "AAA" || !n++'

(or for a regexp pattern: awk '!/^AAA$/ || !n++')

a shorthand for:

awk '! (&0 == "AAA" && count > 0) {print; count++}'
  • I like that awk shorthand. Took a second to parse this early in the day, but I've filed that away for future use.
    – DopeGhoti
    Aug 25 '17 at 15:59
  • +1 for the genius sed solution
    – Philippos
    Aug 25 '17 at 16:32
  • I'm trying to wrap my head around your sed solution. It does work, but how does it know how to delete only the AAA line?
    – neXus
    Aug 28 '17 at 11:11
  • @neXus, see if the new edit makes it clearer. Aug 28 '17 at 11:14
  • Yes, it does. Thank you. I didn't know that the empty //d pattern reused the previous one. I assumed it deleted everything. This makes much more sense. Beautiful solution!
    – neXus
    Aug 28 '17 at 11:18

Stéphane Chazelas' awk solution is beautiful:

awk '!/AAA/ || !n++' file.in

This may be generalized as

awk '$0 !~ pattern || !n++' pattern="$pattern" file.in

for a given shell variable $pattern containing some regular expression.

If $pattern contains backslashes, these need to be escaped (\\), or you may use

P="$pattern" awk '$0 !~ ENVIRON["P"] || !n++' file.in

Just exchange buffers each time you encounter that line, if the pattern space contains the same line delete it, else retrieve the line from the hold buffer:

sed -e '/^AAA$/{x;//d;g' -e'}' infile


sed '/^AAA$/{
}' infile

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.