3

This is a sed-specific question; I am well aware it could be done with other tools but I am working on expanding my knowledge of sed.

How can I use sed to globally quote (actually backtick) a word that is not specified in the script? The word is held in the hold space.

What I want is something like:

s/word/`&`/g

But the trick is, word will be contained not in the sed script but in the hold space. So it looks something more like:

H
g
s/^\(.*\)\n\(.*\)\1\(.*\)$/\2`\1`\3/

which will quote one occurrence of the word held in the hold space. I want to quote all of them, but I can't just add a g flag, because of the way this uses backreferences rather than a static regex.

H
g
s/^\(.*\)\n\(.*\)\1\(.*\)\1\(.*\)$/\2`\1`\3`\1`\4/

This handles two occurrences of the word, but fails on one, and ignores more than one.

I thought I could use something clean and simple like:

s//`&`/g

But that reuses the last used regex, not what it matches. (Which makes sense.)

Is there any way in sed to do what I am trying to do? (Actually I would be interested in seeing how easy this would be in perl, but I would still like to see how to do it in sed.)

UPDATE

Not that it's needed for this question, but I thought I would give a little more context on what exactly I was doing when I came up with this question:

I had a big text file of documentation, certain parts of which needed to be condensed and summarized into an asciidoc table. It was pretty easy because of the Description: and Prototype: lines, etc., so I actually wrote a quick sed script to do all the parsing for me. It worked beautifully—but the one thing it was missing was that I wanted to backtick the words in the Description line that matched the arguments listed in the Prototype line. The prototype lines looked something like this:

Prototype: some_words_here(and, arg, list,here)

There were upwards of 200 different entries in the table I was outputting (and the source documentation included a lot more text than that) and each arglist only needed to be used to backtick-quote matching words on a single line. To make things trickier, some of the args were not in the Description line, some were in more than once, and some arglists were empty().

However, given that sometimes an arg would match a part of a word, which I didn't want to get backticked, and sometimes an arg name was a common word (like from) which I only wanted to get backticked when it was used in the context of explaining the use of the function, an automated solution wasn't actually a good fit at all and I instead used vim to do the job semi-manually, with the help of some tricky macros. :)

Improve this question

2 Answers 2

Reset to default
6

That was a hard one. Assuming you have a file like this:

$ cat file
word
line with a word and words and wording wordy words.

Where:

  • Line 1: is the search pattern that should be held in the hold space and quoted to `word`.
  • Line 2: is the line to seach and replace globally.

The sed command:

sed -n '1h; 2{x;G;:l;s/^\([^\n]\+\)\n\(.*[^`]\)\1\([^`]\)/\1\n\2`\1`\3/;tl;p}' file

Explanation:

  • 1h; save the first line to the hold space (this is wait we want to search for).
    • hold space contains: word
  • 2{...} applies to the second line.
  • x; exchange the pattern space and the hold space.
  • G; append the hold space to the pattern space. In the pattern space we have now: 
word # I will call this line the "pattern line" from now on
line with a word and words and wording wordy words.
  • :l; set a label called l as point for later.
  • s/// do the actual search/replace in the pattern space mentioned above:
    • ^\([^\n]\+\)\n search in the "pattern line" for all characters (from the beginning of the line ^) which are not a newline [^\n] (one or more times \+), until a newline \n. This is now stored in the back-reference \1. It contains the "pattern line".
    • (.*[^`]) search for any character .* followed by a character, which is not a backtick [^`]. This is stored in \2. \2 contains now: line with a word and words and wording wordy, until the last occurence of word, because...
    • \1 is the next search term (the back-reference \1, word), hence what the "pattern line" contains.
    • ([^`]) this is followed by another character which is not a backtick; saved to reference \3. If we don't do this (and the part in \2 from above), we would end of in an endless loop quoting the same word, again and again -> ````word````, because s/// would always be successful and tl; jumps back to :l (see tl; further down).
    • \1\n\2\1\3 all of the above is replaced by the back-references. The second \1 is the one we should quote (note the first reference is the "pattern line").
  • tl; if the s/// was successful (we replaced something) jump to the label called l and start again until there is nothing more to search and replace. This is the case, when all occurences of word are replaced/quoted.
  • p; when all is done, print the altered line (pattern space).

The output:

$ sed -n '1h; 2{x;G;:l;s/^\([^\n]\+\)\n\(.*[^`]\)\1\([^`]\)/\1\n\2`\1`\3/;tl;p}' file
word
line with a `word` and `word`s and `word`ing `word`y `word`s.
Improve this answer
3
  • Brilliant! The trick of using [^`] is great. Is there a reason you've backslashed the backticks in your character classes?
    – Wildcard
    Commented Oct 28, 2015 at 9:51
  • 1
    @Wildcard You're right, the backslash is not necessary. It was probably my "auto-escape" behaviour.
    – chaos
    Commented Oct 28, 2015 at 9:54
  • Looking at this again now, two months later. I still think it's brilliant but think it's fair to note the caveats I've noticed: (1) if the second line starts with the pattern line (word in your example) it will not be matched because of [^`]. (2) If the line ends with the pattern line it will not be matched for the same reason. Still—using a recursive sed solution is brilliant and I recently used a similar concept to write a clisp-history-cleaner-upper in sed: stackoverflow.com/a/34283441/5419599
    – Wildcard
    Commented Dec 17, 2015 at 3:51
3

Lookup tables can be difficult - and expensive - because you have to search both ends of the pattern space simultaneously. It can, at least, be implemented more or less straightforwardly, though. You have to consider that no matter what you do, you can only reliably handle a single match at a time, and so you might as well give up any hope of a global result here. It will only confuse things anyway - you're not working with a compiled expression, you're literally working with side-effects and both sides to boot.

printf  %s\\n some words to match \
        'and some words and some more words to match them against' |
sed  -ne'$!{H;d;}' -e'G;s/\(\n\).*/\1&\1/;tm' -e:m \
     -e 's/\(.\)\(.*\)\(.*\n\n.*\n\1\2\(\n\)\)/`\1\4\2`\3/;tm'

That's the main loop. It doesn't actually work quite yet because I don't clean it up yet there, but it solves the fundamental problem. Because you have to loop over the same pattern space repeatedly, how can you be sure your match doesn't match twice, right? If you bookend it with some delimiter you'll still match again, and you'll just stack the bookends ad infinitum.

The solution I use here is to mangle the match. I insert a newline after the first character of the match, which I still need to clean up, of course, and which I'll handle a moment. This still doesn't work, though, if your lookup tables can contain members which are subsets of other members, or if you're working with single character sets. There are ways to do that - and better ways to do this - and I will offer you some alternatives if you ask for them.

Here's a little more of it:

printf  %s\\n some words to match \
        'and some words and some more words to match them against' |
sed  -ne'$!{H;d;}' -e'G;s/\(\n\).*/\1&\1/;tm' -e:m \
     -e 's/\(.\)\(.*\)\(.*\n\n.*\n\1\2\(\n\)\)/`\1\4\2`\3/;tm' \
     -e  l

and `s\nome` `w\nords` and `s\nome` more `w\nords` `t\no` `m\natch` \
them against\n\n\nsome\nwords\nto\nmatch\n$

And the cleanup is easy, of course:

printf  %s\\n some words to match \
        'and some words and some more words to match them against' |
sed  -ne'$!{H;d;}' -e'G;s/\(\n\).*/\1&\1/;tm' -e:m \
     -e 's/\(.\)\(.*\)\(.*\n\n.*\n\1\2\(\n\)\)/`\1\4\2`\3/;tm' \
     -e 's/\(`.\)\n/\1/g;P'

and `some` `words` and `some` more `words` `to` `match` them against

That, at least, you can do globally.

My preferred method of doing this sort of thing is to actually build a script for it.

printf  %s\\n some words to match \
        'and some words and some more words to match them against' |
{   sed -e"$(
        sed -ne'$w /dev/fd/3' -e$\q     \
             -e 's/[]\^$/.*[]/\\&/g'    \
             -e 's|..*|s/&/`\&`/g|p'
    )"  <&3
}   3<<""    3<>/dev/fd/3

and `some` `words` and `some` more `words` `to` `match` them against

The sed within the command substitution writes out a sed s///ubstitution statement after taking care to escape any metacharacters any input line but the last might contain. The last line it writes literally to the shared here-doc file descriptor for the outer sed to read as input. The inner sed prints a script that works like:

sed -e's/some/`&`/g'  \
    -e's/words/`&`/g' \
    -e's/to/`&`/g'    \
    -e's/match/`&`/g'

...and hands off the last line to the other sed to handle afterward.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .