50

I cannot seem to make it work. GNU sed documentation says to escape the pipe, but that doesn't work, nor does using a straight pipe without the escape. Adding parens makes no difference.

$ echo 'cat
dog
pear
banana
cat
dog' | sed 's/cat|dog/Bear/g'
cat
dog
pear
banana
cat
dog

$ echo 'cat
dog
pear
banana
cat
dog' | sed 's/cat\|dog/Bear/g'
cat
dog
pear
banana
cat
dog
65

By default sed uses POSIX Basic Regular Expressions, which don't include the | alternation operator. You can switch it into using Extended Regular Expressions, which do include | alternation, with -E (or -r in some older versions of some implementations). You can use:

echo 'cat dog pear banana cat dog' | sed -E -e 's/cat|dog/Bear/g'

and it will work on compliant systems. (-e optionally marks the sed script itself - you can leave it out, it just guards against some kinds of mistake)

Portability to very old seds is complicated, but you can also switch to awk if you need it, which uses EREs everywhere.

5
  • The three BSDs you mentioned all support the -r option as a synonym of -E for compatibility with GNU sed. OpenBSD's and OS X's sed -E will interpret the escaped pipe as a literal pipe, not as alternation operator. Here's a working link to the NetBSD man page and here's one for OpenBSD that isn't ten years old.
    – damien
    Jul 19 '14 at 3:08
  • GNU sed supports -E gnu.org/software/sed/manual/sed.html#index-_002dE.
    – ImHere
    Oct 30 '19 at 20:40
  • 1
    I gather that -E tells it to use extended regular expressions; what does -e do here? For me on MSYS2/Windows and MacOS Catalina, omitting -e works and produces the same output . . . Jul 17 '20 at 0:34
  • 1
    The back references do not work (\1, \2 etc) when using -E. I can't use sed -E -e"s/\(cat|dog\)/\1 - Bear/g". Is there any workaround for that?
    – Joe Jobs
    Jul 18 '21 at 6:25
13

This happens because (a|b) is an extended regular expression, not a Basic Regular Expression. Use the -E option to deal with this.

echo 'cat
dog
pear
banana
cat
dog'|sed -E 's/cat|dog/Bear/g'

From the sed man page:

 -E      Interpret regular expressions as extended (modern) regular
         expressions rather than basic regular expressions (BRE's).

Note that -r is another flag for the same thing, but -E is more portable and will even be in the next version of POSIX specifications.

11

The portable way to do this - and the more efficient way - is with addresses. You can do this:

printf %s\\n cat dog pear banana cat dog |
sed -e '/cat/!{/dog/!b' -e '};cBear'

In this way if the line does not contain the string cat and does not contain the string dog sed branches out of the script, autoprints its current line and pulls in the next to begin the next cycle. It therefore does not perform the next instruction - which in this example changes the entire line to read Bear but it could do anything.

It's probably worth noting also that any statement following the !b in that sed command can only match on a line containing either the string dog or cat - so you can perform further tests without any danger of matching a line that doesn't - which means you can now apply rules to only one or the other as well.

But that's next. Here's output from the above command:

###OUTPUT###
Bear
Bear
pear
banana
Bear
Bear

You can also portably implement a lookup table with backreferences.

printf %s\\n cat dog pear banana cat dog |
sed '1{x;s/^/ cat dog /;x
};G;s/^\(.*\)\n.* \1 .*/Bear/;P;d'

It's a lot more work to setup for this simple example case, but it can make for much more flexible sed scripts in the long run.

In the first line I exchange hold space and pattern space then insert the string <space>cat<space>dog<space> into hold space before exchanging them back.

From then on and on every following line I Get hold space appended to pattern space, then check to see if all of the characters from the beginning of the line until the newline I just added at the end match a string surrounded by spaces after it. If so I replace the entire lot with Bear and if not there is no harm done because I next Print only up to the first occurring newline in pattern space then delete it all.

###OUTPUT###
Bear
Bear
pear
banana
Bear
Bear

And when I say flexible, I mean it. Here it is replacing cat with BrownBear and dog with BlackBear:

printf %s\\n cat dog pear banana cat dog |
sed '1{x;s/^/ 1cat Brown 2dog Black /;x
};G;s/^\(.*\)\n.* [0-9]\1 \([^ ]*\) .*/\2Bear/;P;d'

###OUTPUT###
BrownBear
BlackBear
pear
banana
BrownBear
BlackBear

You can of course expand a great deal on the contents of the lookup table - I picked up the idea from Greg Ubben's usenet emails on the subject when, in the 90's, he described how he constructed a crude calculator out of a single sed s/// statement.

2
  • 1
    phew, +1. You have a penchant for thinking out of the box I must say
    – iruvar
    Jul 19 '14 at 4:52
  • 1
    @1_CR - See my last edit - not my idea - which is not to say that I don't appreciate that and consider it a compliment. But I like to give credit where it's due.
    – mikeserv
    Jul 19 '14 at 5:04
1

this is a pretty old question, but in case someone wants to try, there is a fairly low effort way to do this in sed with sed files. Each option can be listed on a separate line, and sed will evaluate each one. It's a logical equivalent of or. For example, to remove lines that contain a certain code:

you can say : sed -E '/^\/\*!(40103|40101|40111).*\/;$/d'

or put this in your sed file:

/^\/\*!40103.*\/;$/d
/^\/\*!40101.*\/;$/d
/^\/\*!40111.*\/;$/d
1

Here's a technique that does not make use of any implementation specific options to sed (e.g. -E, -r). Instead of describing the pattern as a single regex cat|dog, we can simply run sed twice:

echo 'cat
dog
pear
banana
cat
dog' | sed 's/cat/Bear/g' | sed 's/dog/Bear/g'

It's an obvious workaround really, but worth sharing. It naturally generalizes to more than two pattern strings, though a very long chain of sed's isn't too good looking.

I often use sed -i (which works the same in all implementations) to make changes in files. Here, a long list of pattern strings may be incorporated nicely, as each temporary result is save to the file:

for pattern in cat dog owl; do
    sed -i "s/${pattern}/Bear/g" myfile
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.