2

So I came to know about the existence of buffer in sed (pattern buffer, hold buffer, etc) and thought if there was an easy way to hold/save or just reuse the result of a sed command (eg: substitution) for another command/sed invocation?

Say I'm doing substitution from the output of a pipe:

somecommand | sed 's/somepattern/somethingelse/g'

And i want to somehow reuse the output of the substitution above, how could i do it? I know i could just use:

  1. A tempfile (eg: using primitive mv + echo etc)
  2. Another set of pipe, sed invocation and xargs (eg: sed ... | xargs -I{} sed ...)

But both of the above would prevent me from only using a single invocation of sed (which is what I want to do here).

If there was such a way known to me, I would then just do:

somecommand | sed -i 's/somepattern/somethingelse/g;s/[reusing result from last substitution]/someotherthings/g' file

Where the part between [] would be reusing the result of the substitution as pattern or for something else...

I did try and look up any possible solutions on the sed documentation but there wasn't much example if any leading to this.

Any feedback/answer appreciated.

P.S.: The output of the examples above isn't really the focus here, but to make things clearer, the somecommand could be replaced to this:

echo "hello"

Literally.

Here another thing i tried, that will serve as a more understandable example:

#!/bin/sh
echo -e "hello\nworld" | sed -n '
l #enable debugging
'/hello/' { # match the string from the echo pipe/command output
    s/hello/test/ #do a substitution
    p #print
    x # keep in
}'

I managed to do the first half; Now I need to know how to use what reside in x so i could, say, use it for another substitution (or any other sed invocation/action/command)...

Here what I tried (but this time failed):

#!/bin/sh
echo -e "hello\nworld" | sed -n '
l #enable debugging
'/hello/' { # match the string from the echo pipe/command output
    s/hello/test/ #do a substitution
    p #print
    x # keep in
    s/x/somethingelse/ # <---- what is failing
}'

This doesn't work. I'm guessing there a way to get whatever content is in x here but I don't know it.

18
  • 3
    Can you be more specific and providw a concrete input/expected output set?
    – guest_7
    May 20 at 4:03
  • there isn't any? the somecommand just output an example string of small lenght (length here doesn't matter much), then some action is done by the first sed command (here I took the liberty to make it substitute to some example pattern) then reuse the output of the previous sed command, but as pattern for another actino (eg: another substitution) and as seen on the very last example, and for more clarity, done on a file... @guest_7 May 20 at 4:11
  • The main thing I want to know here is mainly how i could easily save the output of a sed command, without using the two method i described in my post and reuse it as a pattern for another set of sed command :) (the documentation didn't help much in that regard) @guest_7 May 20 at 4:12
  • when using sed 's/pattern1/replace1/g;s/pattern2/replace2/g' sed will remplace pattern1 by replace1, then replace1 is discarded from sed's memory but, of course, keep in output file. There is no way to reuse replace1 inside pattern2
    – Archemar
    May 20 at 4:39
  • Yes, but I don't want to do (following your example) sed 's/pattern1/replace1/g;s/pattern2/replace2/g' but instead sed 's/pattern1/replace1/g;s/[reuse result of last substitution]/replace2/g'...and I know it's possible because I seen some command hold/keep result from one sed invocation into a buffer, and use it for other command...but the documentation isn't the best in showing clear example of this @Archemar May 20 at 4:46
2

As far as I understand you, you want some kind of dynamic search pattern by using the output of a past substitution as pattern for another replacement.

I find it helpful to work with examples. Given a file like

green2
gold1
blue3
gold2
red4
more gold2 to find

Now you want to replace each 1 by a 2 and store the resulting line to do something when it occurs again. In this example file you would want to replace gold2 but you can't know, because in another file it could be different. The output is suppose to be

green2
gold2
blue3
replace
red4
more replace to replace

There can be similar real world tasks and you usually handle those in sed like this:

sed -e 's/1/2/;tfound' -e 'G;s/\(..*\)\(.*\)\n\1$/replace\2/;P;d' -e ':found' -e h inputfile

The concept is to store the resulting line in hold buffer and use a backreference to match each line against the hold buffer. In detail:

  • s/1/2/ is the obvious part: you replace a 1 by 2
  • tfound means to branch to mark :found in case a replacement was made. In this case, the line is stored in the hold space with h and the line with the replacement is printed (if you don't want to print it, you could add a delete)
  • Now comes the part for lines to check for the occurence of our hold space pattern: G appends the hold space to the current pattern space, so the pattern space consists of
  • s/\(..*\)\(.*\)\n\1$/replace\2/ forms two groups in the current line: The first one is repeated as \1 after the newline, so this is the dynamic pattern in hold space (note that ..* requires the pattern to be at least one char, so we avoid an empty hold space to match); the second one is the rest of the line that must not be deleted, so we recylce it as \2 in the replacement
  • We could print the line, if a replacement was made, but if there was no replacement, we have to remove the appended stuff. We could do so by s/\n.*//, but we can also use the P command to only print the first line, then delete to avoid default output.

This is limited to replacing only one occurence of the dynamic search pattern in the line, but you can easily add a loop to make it work for multiple replacements.

Update: The OP clarified, the second stage should also be applied to the original matching line and all following lines (until a new match), so

hello
world
and test it

is supposed to become

replace world and replace it

In this case you use the same mechanism with different glue logic:

sed -ne '/hello/{s//test/;h;}' -e 'G;s/\(..*\)\(.*\)\n\1$/replace\2/;P'

Option -n suppresses all default output, because all desired output is done by P command now. For the matching string (hello), the substitution is performed (empty pattern means to reuse the last pattern) and place it in hold buffer, then execute the following commands, so next substitution is done in the same line as well.

Update 2: In the linked example, the hello lines should not be modified at all. You can do this with a little modification:

sed -ne '/hello/{h;s//test/;x;}' -e 'G;s/\(..*\)\(.*\)\n\1$/replace\2/;P'
9
  • btw, It sadly doesn't work for the second substitution (using the example in my post with "hello\nworld"). May 21 at 6:52
  • Maybe because I didn't understand the supposed output of your example. You change hello to test, but there is no test in your file. Do you also want to apply the second stage to the original line (in my example the second line should also be replace)? You can easily modify the logic of my script: sed -e 's/hello/test/;Tskip' -e h -e ':skip' -e 'G;s/\(..*\)\(.*\)\n\1$/somethingelse\2/;P;d' inputfile
    – Philippos
    May 21 at 7:03
  • Or using a command group instead of a branch: sed -e '/hello/{s//test/;h;}' -e 'G;s/\(..*\)\(.*\)\n\1$/replace\2/;P;d'. If this is what you actually expect, I can include it in the answer.
    – Philippos
    May 21 at 7:05
  • Yeah, would love if this was included in your answer :D Thanks for greatly detailing everything like this, this help a lots. Also, could you take into account if the inputfile contain the (following above example) the match "test" after the first substitution? so it can replace it on the second substitution but on the actual file this time May 21 at 7:15
  • Just seen your edit; this time it does work great and both substitution work as expected, only...it seems it does the two substitution on both the file and the stdin output. see this for more context. May 21 at 9:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.