I found a one-line sed command on this site which removes duplicate entries in $PATH. There is one concept that I can't get my mind around. It uses saved matches in the match section to detect the duplicate, and then replaces the match with \1\2.

In the example below, I can't understand why \1 is not always 1111. The test case obviously shows that the \1 match progresses further down the pattern space, but I can't understand why.

sed script (sed_cmd):

p  #debug
s/[yx]//g   #debug remove the field indicators for the next pass
s/;/:/g     #debug

Test command + output:

echo "0000:1111:2222:3333:4444:1111:2222:3333:0000" | sed -f sed_cmd

The pattern in the substitute command is: [:;]\([^:;]*\)\([:;].*\)[;:]\1. Notice the \1 on the end. This means whatever text matches the first group, \([^:;]*\) must also occur at the end of the pattern.

Your pattern space is initially 0000:1111:2222:3333:4444:1111:2222:3333:0000. The pattern matches :1111:2222:3333:4444:1111, and 1111 matches the first group and also the \1 at the end.

After the first round of substitutions, the pattern space has been changed to 0000:1111:2222:3333:4444:2222:3333:0000. Now, if the 1111 in the pattern space is used to match \([^:;]*\), there is no 1111 left to match \1. Thus, the pattern fails to match and the regex engine tries something else. In this case, using the next available match for the first group, 2222, works.

  • Not sure I agree. I placed the "y" and "x" characters into the replacement pattern so I could see what was actually the \1 & \2 matches. the \1 match is always just the digits (no colon). You can see on the 2nd and 3rd pass, the "y" surrounds a different set...and I don't see what makes the match move down the string if the s command always starts at the befinning of the pattern space. I still don't get it. – dpinvidic Jul 10 '14 at 18:17
  • First, thanks for your help. The first part of the match pattern is /:([^:;].*). So I feel that this should always match ":1111" on every repeat of the substitute command. I don't see how it matches ":2222" on the second pass. – dpinvidic Jul 10 '14 at 18:30
  • @dpinvidic :1111 and :2222 are very similar. You should be able to easily see that the pattern matches both strings. Your question should not be "why does this pattern match :2222", because that should be obvious (it's almost the same string as :1111), but rather "why does this pattern NOT match :1111 when :1111 comes before :2222". I explained that in my answer. I have edited my answer a bit. Please reread the entire answer carefully. – jw013 Jul 10 '14 at 18:39
  • Darn...I knew it was something simple. I just forgot the basic fact that SED started each pattern match attempt with one character. If the match fails, it moves to the next character and once again attempts to match the full expression. I had it in my mind that the very first ":1111" was always a match and became \1. Thank You so much....I feel like a Dumb S..t – dpinvidic Jul 10 '14 at 18:58
  • @jw013 - congrats on 20k! – slm Oct 24 '14 at 13:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.