5

I've learned that, with sed, it's possible to match the nth occurrence of a pattern by using the null data mode (-z): sed -z 's/foo/bar/2'.

How do I match beginning of lines when using -z?

If I execute:

echo $'foo\nfoo\nfoo' | sed -z 's/^foo/baz/2'

there are no replacements, because ^ indicates the beginning of the whole string:

$ echo $'foo\nfoo\nfoo' | sed -z 's/^foo/baz/g'
baz
foo
foo

In Perl, there is the m regex modifier (when slurping), but in sed it doesn't help.

4 Answers 4

7
echo $'foo\nfoo\nfoo' | sed -Ez 's/(^|\n)foo/\1baz/2'

^ is needed to properly count foo that happens to be in the beginning of the first line.

Note the code counts occurrences of (^|\n)foo, not foo. If you want to count foos, but replace only when the desired occurrence happens to be in the beginning of a line, then this code is not a solution. For example in:

echo $'foo foo\nfoo foo\nfoo foo' | sed -Ez 's/(^|\n)foo/\1baz/3'

the foo being replaced is not the third foo.

Tested with GNU sed 4.8.

1
  • the foo being replaced is not the third foo Yes, it is It is the third foo that is at the start of a line in the pattern space. Commented Aug 10, 2022 at 1:25
3

Is not that simple as the selected answer seems to make it. First, what is the 2 (or any other number in s///2) actual meaning? It simply means that the second regex match on the line where it is applied will be changed.

In

$ printf '%s\n' 'foo foo foo' 'foo foo foo' 'foo foo foo' | sed 's/foo/bar/2'
foo bar foo
foo bar foo
foo bar foo

The sed command changed the second (2) instance of foo with bar but for each line that it got applied to (all of them). It could be changed to work only on one line or a range of lines:

printf '%printf '%s\n' 'foo foo foo'{,,,,} | sed '3,4s/foo/bar/2'
foo foo foo
foo foo foo
foo bar foo
foo bar foo
foo foo foo

Note that only lines 3 and 4 got changed, not all, and that in all those lines, the instance of foo that got changed was the second (2).

That is the way that the s/foo/bar/2 works.

-z

Now, if -z is used, lines end with \0 (not \n). But the replacement works in the exact same way (with \0 instead of \n):

$ printf '%s\0' 'foo foo foo'{,,,,} | sed -z '3,4s/foo/bar/2' | xxd
00000000: 666f 6f20 666f 6f20 666f 6f00 666f 6f20  foo foo foo.foo 
00000010: 666f 6f20 666f 6f00 666f 6f20 6261 7220  foo foo.foo bar 
00000020: 666f 6f00 666f 6f20 6261 7220 666f 6f00  foo.foo bar foo.
00000030: 666f 6f20 666f 6f20 666f 6f00            foo foo foo.

mixed \0 and \n

In echo $'foo\nfoo\nfoo' | sed -z 's/^foo/baz/2' there are not enough foo in each line to make it possible to change the second, but in this next example: should it be?

$ printf 'foo foo foo\nfoo foo foo\n' | sed -z 's/^foo/baz/2'
foo foo foo
foo foo foo

Oops, no there are not enough foo at the start of the line either. The problem is: where does a line start? at a newline or at a \0, or both?, or none?

It makes no sense to think of "start of the line" with ^ when using '-z'.

That is an internal confusion for sed. Remember: the use of -z is experimental and might lead to odd problems.

pattern space

In fact, for the substitution to work correctly, the whole input needs to be in the pattern space. No, it doesn't work if the input has NULs (\0), those would be treated as line delimiter (or, in awk parlance, as record separators).

$ printf 'foo\0foo\0foo\0' | sed -z 's/^foo/baz/2'
foofoofoo

We can get the whole input file inside sed pattern space by using H;1h;$!d;x;..... and then try the ^foo substitution:

$ printf 'foo\0foo\0foo\0\n' | sed -z 'H;1h;$!d;x;l;s/^foo/ baz /M2'
foo\000foo\000foo\000\n$foo baz foo

The l allow us to see what is inside the pattern space, and the M flag is required so that the ^ is matching more that the first line. If the M is not used, the ^foo will match only on the first line (at the start of the pattern space).

An alternative to M is:

$ printf 'foo\0foo\0foo\0' | sed -z 'H;1h;$!d;x;l;s/\(^\|\x0\)foo/ baz /2'
foo\000foo\000foo$foo baz foo

Please note that the trailing \0 got removed, what got into the internal pattern space was foo\000foo\000foo, which is missing the trailing \0, which was clearly supplied on the input.

We can get all three \0 by adding a trailing newline:

$ printf 'foo\0foo\0foo\0\n' | sed -z 'H;1h;$!d;x;l;s/\(^\|\x0\)foo/ baz /2'
foo\000foo\000foo\000\n$foo baz foo

Which is a clear demonstration that sed treats a \0 as the delimiter sometimes, and a \n as the delimiter on other cases.

In short, sed with the -z option is still experimental.

0
2

In Perl, there is the m regex modifier (when slurping), but it doesn't help.

Sure it does.

printf '%s\n' foo foo foo |\
perl -0777 -pe 's/^(foo)/++$c == 2 ? "bar" : $1/egm'

We slurp with -0777, match a bunch of times g and with m helping that so ^ matches within the slurp, and then e eval the bar in only when a counter variable is 2.

1
  • Sorry, I've realized that my comment was misleading! What I meant it that it doesn't help in sed.
    – Marcus
    Commented Aug 8, 2022 at 22:21
2

In the slurp mode (-z) , GNU sed treats the record separator as the null value. But since in an ASCII text file there are no null characters the whole file is essentially one record or line to sed. To work around this we first change all newlines (\n) to the line separator (NUL) and then apply the s/// in t Multiline mode on the second match. Finally reverse transform

printf '%s\n' foo foo foo |
sed -z '
  y/\n/\x00/
  s/^foo/BAR/M2
  y/\x00/\n/
'

Output:

foo
BAR
foo
7
  • Why printf 'foo\0foo\0foo\0' | sed -z 's/^foo/BAR/2' print foofoofoo, then ? Commented Aug 9, 2022 at 3:31
  • @QuartzCristal because the lines (separated by \0) donot have a second foo
    – guest_7
    Commented Aug 9, 2022 at 3:44
  • But foo\0foo\0foo\0 is exactly the same as from your example (after the y///). Commented Aug 9, 2022 at 12:09
  • @QuartzCristal actually in my case it's a multi line pattern space, whereas here sed sees three separate lines. The lines are NUL separated REMEMBER
    – guest_7
    Commented Aug 9, 2022 at 13:17
  • sight .... Yes. i do know that the lines are NUL separated, And that is exactly what was provided as input: foo\0. That is a foo followed by a NUL remember?. But if you want to try multiline, just add the option, here it is for your review: printf 'foo\0foo\0foo\0' | sed -z 's/^foo/BAR/M2' which, as I already said, prints foofoofoo . Commented Aug 10, 2022 at 0:47

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .