I want to use sed to replace anything in a string between the first AB and the first occurrence of AC (inclusive) with XXX.

For example, I have this string (this string is for a test only):


and I would like output similar to this: ssXXXABnnACss.

I did this with perl:

$ echo 'ssABteAstACABnnACss' | perl -pe 's/AB.*?AC/XXX/'

but I want to implement it with sed. The following (using the Perl-compatible regex) does not work:

$ echo 'ssABteAstACABnnACss' | sed -re 's/AB.*?AC/XXX/'
  • 2
    This doesn't make sense. You have a working solution in Perl, but you want to use Sed, why?
    – Kusalananda
    Jul 23 '16 at 6:44

Sed regexes match the longest match. Sed has no equivalent of non-greedy.

What we want to do is match

  1. AB,
    followed by
  2. any amount of anything other than AC,
    followed by
  3. AC

Unfortunately, sed can’t do #2 — at least not for a multi-character regular expression.  Of course, for a single-character regular expression such as @ (or even [123]), we can do [^@]* or [^123]*.  And so we can work around sed’s limitations by changing all occurrences of AC to @ and then searching for

  1. AB,
    followed by
  2. any number of anything other than @,
    followed by
  3. @

like this:

sed 's/AC/@/g; s/AB[^@]*@/XXX/; s/@/AC/g'

The last part changes unmatched instances of @ back to AC.

But this is a reckless approach because the input could already contain @ characters. So, by matching them, we could get false positives.  However, since no shell variable will ever have a NUL (\x00) character in it, NUL is likely a good character to use in the above work-around instead of @:

$ echo 'ssABteAstACABnnACss' | sed 's/AC/\x00/g; s/AB[^\x00]*\x00/XXX/; s/\x00/AC/g'

The use of NUL requires GNU sed. (To make sure that GNU features are enabled, the user must not have set the shell variable POSIXLY_CORRECT.)

If you are using sed with GNU's -z flag to handle NUL-separated input, such as the output of find ... -print0, then NUL will not be in the pattern space and NUL is a good choice for the substitution here.

Although NUL cannot be in a bash variable it is possible to include it in a printf command. If your input string can contain any character at all, including NUL, then see Stéphane Chazelas' answer which adds a clever escaping method.

  • I just edited your answer to add a lengthy explanation; feel free to trim it or roll it back. Jul 23 '16 at 5:33
  • @G-Man That is an excellent explanation! Very nicely done. Thank you.
    – John1024
    Jul 23 '16 at 6:51
  • You can echo or printf an `\000' just fine in bash (or the input could come from a file). But in general, a string of text is of course not likely have NULs.
    – ilkkachu
    Jul 23 '16 at 14:39
  • @ilkkachu You are right about that. What I should have written is that no shell variable or parameter can contain NULs. Answer updated.
    – John1024
    Jul 23 '16 at 19:24
  • Wouldn't this be a whole lot safer if you changed AC to AC@ and back again? Jul 25 '16 at 8:52

sed - non greedy matching by Christoph Sieghart

The trick to get non greedy matching in sed is to match all characters excluding the one that terminates the match. I know, a no-brainer, but I wasted precious minutes on it and shell scripts should be, after all, quick and easy. So in case somebody else might need it:

Greedy matching

% echo "<b>foo</b>bar" | sed 's/<.*>//g'

Non greedy matching

% echo "<b>foo</b>bar" | sed 's/<[^>]*>//g'

  • 6
    The term “no-brainer” is ambiguous.  In this case, it is not clear that you (or Christoph Sieghart) thought this through.  In particular, it would have been nice if you had showed how to solve the specific problem in the question (where the zero-of-more-of- expression is followed by more than one character).  You may find that this answer doesn’t work well in that case.
    – Scott
    Oct 12 '17 at 22:14
  • The rabbit hole is much deeper than it seemed to me at first glance. You are right, that workaround doesn't work well for multi-character regular expression.
    – gresolio
    Oct 15 '17 at 20:15

Some sed implementations have support for that. ssed has a PCRE mode:

ssed -R 's/AB.*?AC/XXX/'

AT&T ast sed supports the *? operator as a non-greedy version of * in its extended (with -E) and augmented (with -A regexps).

sed -E 's/AB.*?AC/XXX/'
sed -A 's/AB.*?AC/XXX/'

In that implementation and those -E/-A modes, more generally, perl-like regexps can be used inside (?P:perl-like regexp here), though as seen above, it's not necessary for the *? operator.

Its augmented regexps also have conjunction and negation operators:

sed -A 's/AB(.*&(.*AC.*)!)AC/XXX/'

Portably, you can use this technique: replace the end string (here AC) with a single character that doesn't occur in either the beginning or end string (like : here) so you can do s/AB[^:]*://, and in case that character may appear in the input, use an escaping mechanism that doesn't clash with the begin and end strings.

An example:

sed 's/_/_u/g; # use _ as the escape character, escape it
     s/:/_c/g; # escape our replacement character
     s/AC/:/g; # replace the end string
     s/AB[^:]*:/XXX/; # actual replacement
     s/:/AC/g; # restore the remaining end strings
     s/_c/:/g; # revert escaping

With GNU sed, an approach is to use newline as the replacement character. Because sed processes one line at a time, newline never occurs in the pattern space, so one can do:

sed 's/AC/\n/g;s/AB[^\n]*\n/XXX/;s/\n/AC/g'

That generally doesn't work with other sed implementations because they don't support [^\n]. With GNU sed you have to make sure that POSIX compatibility is not enabled (like with the POSIXLY_CORRECT environment variable).


No, sed regexes don't have non-greedy matching.

You can match all text up to the first occurrence of AC by using “anything not containing AC” followed by AC, which does the same as Perl's .*?AC. The thing is, “anything not containing AC” cannot be expressed easily as a regular expression: there is always a regular expression that recognizes the negation of a regular expression, but the negation regex gets complicated fast. And in portable sed, this isn't possible at all, because the negation regex requires grouping an alternation which is present in extended regular expressions (e.g. in awk) but not in portable basic regular expressions. Some versions of sed, such as GNU sed, do have extensions to BRE that make it able to express all possible regular expressions.

sed 's/AB\([^A]*\|A[^C]\)*A*AC/XXX/'

Because of the difficulty of negating a regex, this doesn't generalize well. What you can do instead is to transform the line temporarily. In some sed implementations, you can use newlines as a marker, since they can't appear in an input line (and if you need multiple markers, use newline followed by a varying character).

sed -e 's/AC/\
&/g' -e 's/AB[^\
]*\nAC/XXX/' -e 's/\n//g'

However, beware that backslash-newline doesn't work in a character set with some sed versions. In particular, this doesn't work in GNU sed, which is the sed implementation on non-embedded Linux; in GNU sed you can use \n instead:

sed -e 's/AC/\
&/g' -e 's/AB[^\n]*\nAC/XXX/' -e 's/\n//g'

In this specific case, it's enough to replace the first AC by a newline. The approach I presented above is more general.

A more powerful approach in sed is to save the line into the hold space, remove all but the first “interesting” part of the line, exchange the hold space and the pattern space or append the pattern space to the hold space and repeat. However, if you start doing things that are this complicated, you should really think about switching to awk. Awk doesn't have non-greedy matching either, but you can split a string and save the parts into variables.


The solution is quite simple. .* is greedy, but it is not absolutely greedy. Consider matching ssABteAstACABnnACss against the regexp AB.*AC. The AC that follows .* must actually have a match. The problem is that because .* is greedy, the subsequent AC will match the last AC rather than the first one. .* eats up the first AC while the literal AC in the regexp matches the last one in ssABteAstACABnnACss. To prevent this from happening, simply replace the first AC with something ridiculous to differentiate it from the second one and from anything else.

echo ssABteAstACABnnACss | sed 's/AC/-foobar-/; s/AB.*-foobar-/XXX/'

The greedy .* will now stop at the foot of -foobar- in ssABteAst-foobar-ABnnACss because there is no other -foobar- than this -foobar-, and the regexp -foobar- MUST have a match. The previous problem was that the regexp AC had two matches, but because .* was greedy, the last match for AC was selected. However, with -foobar-, only one match is possible, and this match proves that .* is not absolutely greedy. The bus stop for .* occurs where only one match remains for the rest of the regexp following .*.

Note that this solution will fail if an AC appears before the first AB because the wrong AC will be replaced with -foobar-. For example, after the first sed substitution, ACssABteAstACABnnACss becomes -foobar-ssABteAstACABnnACss; therefore, a match cannot be found against AB.*-foobar-. However, if the sequence is always ...AB...AC...AB...AC..., then this solution will succeed.


One alternative is to change the string so you want the greedy match

echo "ssABtCeCAstACABnnACss" | rev | sed -E "s/(.*)CA.*BA(.*)/\1CA+-+-+-+-BA\2/" | rev

Use rev to reverse the string, reverse your match criteria, use sed in the usual fashion and then reverse the result....


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.