sed's capture grouping ALTERNATION is not working as expected.

While alternation has the lowest precedence, here is the confusion:

echo "abcd_aefghi" | sed -r "s/(a)(b)(c)(d)|(a)(e)(f)(g)/\8/g;"

observed result --> _ghi
expected result --> error: invalid reference \8 RHS

a--bcd | # Logical alternation (Not bitwise!!)
#must be different separated logical state processing!! 
#not involvement of look behind registered reference grouping Counting!!
a--efg ;


sed -r "s/(a)((b)(c)(d)|(e)(f)(g))/\8/;"
observed result --> _ghi

more confusion if

sed -r "s/(a)(b)(c)(d)()|((a)(e)(f)(g))/\8/g"

observed result --> _ehi

also more confusion

sed -r "s/(a)(b)(c)(d)()||||i am Not comment, Whats going here?|||||||((a)(e)(f)(g))/\8/g"

observed result --> _ehi
expected result --> full parsing error, because "||"

worst is for inside grouping matching

echo "aBcB_aCfC" | sed -r "s/(a)((B)(c)(\1)|(C)(f)(\1))/\4/g;"
#Infield twin uppercase's must match!
observed result --> aBcB_aCfC
expected result --> c_f

perl -pe repeats the same problem!

The problem is that references counting go beyond alternation scope, rather than resetting the counting after recognizing the alternation symbol.

sed version 4.2.2 on Fedora 20.

Of course I give here the very generic-basic problem.

The real script is very complicated long text raw-parsing.

My initial goal here was to print the 4th element in each alternation mass grouping. Now I must split the match, which enlarges the code significantly.

Can someone please reduce my confusion?

  • Works fine for me. I'm using GNU sed 4.5. What version are you using? – Sparhawk May 1 '18 at 13:09
  • show your result ?? – mr.tee May 1 '18 at 13:17
  • how about the same issue: sed -r "s/(a)((b)(c)(d)|(e)(f)(g))/\8/;" – mr.tee May 1 '18 at 13:36
  • please show us how to implement with (?|pattern) ? – mr.tee May 1 '18 at 13:39
  • Stéphane Chazelas has already added that in answer... it would help if you added some more sample input and show what is your expected output.. perhaps, something like sed -r 's/(a)(b|e)(c|f)(d|g)/\4/g' or sed -r 's/(a)([be])([cf])([dg])/\4/g' might help you as well? – Sundeep May 1 '18 at 15:38

\x is expanded to what is captured in the xth capture group, capture groups being numbered left to right based on the occurrence of the opening brace in the regexp.

echo "abcd_aefghi" | sed -r "s/(a)(b)(c)(d)|(a)(e)(f)(g)/\8/g;"
                               1  2  3  4   5  6  7  8

That regexp matches twice. Once on abcd (where the 8th capture group captures nothing) and once on aefg, where the 8th capture group captures a g. So abcd is replaced with nothing and aefg with g and _ and hi are left untouched so you get _ghi as expected.


sed -r "s/(a)(b)(c)(d)()|((a)(e)(f)(g))/\8/g"
          1  2  3  4  5  67  8  9  10

You get _ehi because the 8th group is now the (e) one.

sed -r "s/(a)(b)(c)(d)()||||i am Not comment, Whats going here?|||||||((a)(e)(f)(g))/\8/g"
          1  2  3  4  5                                               67  8  9  10

is no different. Except (though that's not visible) that because of the ||, there will be empty matches in between h and i and an extra one at the end (a few more with perl).

resetting numbers based on the alternation operator would not make for a very usable API. If you want to expand to what is matched by the 4th capture group on either side of the alternation on your example, you can always do:

sed -r 's/(a)(b)(c)(d)|(a)(e)(f)(g)/\4\8/g'
          1  2  3  4   5  6  7  8

Which would give you the same as perl's:

perl -lpe 's/(?|(a)(b)(c)(d)|(a)(e)(f)(g))/\4/g'
                1  2  3  4   1  2  3  4

(where \4 could expand to either what (d) or (g) captured).


As I understand you are not happy with the given answer, as it messes your code for your real problem. Hard to help you without knowing that real problem.

To keep the fourth part of each alternation, you should probably do it differently:

sed -r 's/abcd|aefg/\n&/g;s/\n...//g'

marks each match with newline and removes the newline with the three unwanted chars in a second pass (this is GNU-sed only, but you could do it similar for a different sed).

No enlarged code with this approach. Sometimes you just need to do it differently. Of course you can also do this with groups instead of just chars.

But if this doesn't solve your real problem (and you can't figure how to adapt it), tell us your actual problem with example data.

  • Of course I give here the very generic-basic problem. – mr.tee May 3 '18 at 11:59
  • Yes, I understood that. And does my generic solution for your generic problem also help for your actual problem? – Philippos May 3 '18 at 12:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.