Replace multiple strings in a single pass

Question

I'm looking for a way to replace placeholder strings in a template file with concrete values, with common Unix tools (bash, sed, awk, maybe perl). It is important that the replacement is done in a single pass, that is, what is already scanned/replaced must not be considered for another replacement. For example, these two attempts fail:

echo "AB" | awk '{gsub("A","B");gsub("B","A");print}'
>> AA

echo "AB" | sed 's/A/B/g;s/B/A/g'
>> AA

The correct result in this case is of course BA.

In general, the solution should be equivalent to scanning the input left-to-right for a longest match to one of the given replacement strings, and for each match, performing a replacement and continuing from that point on in the input (none of the already read input nor the replacements performed should be considered for matches). Actually, the details don't matter, just that the results of the replacement are never considered for another replacement, in whole or in part.

NOTE I am only looking for correct generic solutions. Please do not propose solutions which fail for certain inputs (input files, search and replace pairs), however unlikely they may seem.

Answer 1

OK, a general solution. The following bash function requires 2k arguments; each pair consists of a placeholder and a replacement. It's up to you to quote the strings appropriately to pass them into the function. If the number of arguments is odd, an implicit empty argument will be added, which will effectively delete occurrences of the last placeholder.

Neither placeholders nor replacements may contain NUL characters, but you may use standard C \-escapes such as \0 if you need NULs (and consequently you are required to write \\ if you want a \).

It requires the standard build tools which should be present on a posix-like system (lex and cc).

replaceholder() {
  local dir=$(mktemp -d)
  ( cd "$dir"
    { printf %s\\n "%option 8bit noyywrap nounput" "%%"
      printf '"%s" {fputs("%s", yyout);}\n' "${@//\"/\\\"}"
      printf %s\\n "%%" "int main(int argc, char** argv) { return yylex(); }"
    } | lex && cc lex.yy.c
  ) && "$dir"/a.out
  rm -fR "$dir"
}

We assume that \ is already escaped if necessary in the arguments but we need to escape double quotes, if present. That's what the second argument to the second printf does. Since the lex default action is ECHO, we don't need to worry about it.

Example run (with timings for the skeptical; it's just a cheap-o commodity laptop):

$ time echo AB | replaceholder A B B A
BA

real    0m0.128s
user    0m0.106s
sys     0m0.042s
$ time printf %s\\n AB{0000..9999} | replaceholder A B B A > /dev/null

real    0m0.118s
user    0m0.117s
sys     0m0.043s

For larger inputs it might be useful to provide an optimization flag to cc, and for current Posix compatibility, it would be better to use c99. An even more ambitious implementation might try to cache the generated executables instead of generating them each time, but they're not exactly expensive to generate.

Edit

If you have tcc, you can avoid the hassle of creating a temporary directory, and enjoy the faster compile time which will help on normal sized inputs:

treplaceholder () { 
  tcc -run <(
  {
    printf %s\\n "%option 8bit noyywrap nounput" "%%"
    printf '"%s" {fputs("%s", yyout);}\n' "${@//\"/\\\"}"
    printf %s\\n "%%" "int main(int argc, char** argv) { return yylex(); }"
  } | lex -t)
}

$ time printf %s\\n AB{0000..9999} | treplaceholder A B B A > /dev/null

real    0m0.039s
user    0m0.041s
sys     0m0.031s

Answer 2

printf 'STRING1STRING1\n\nSTRING2STRING1\nSTRING2\n' |
od -A n -t c -v -w1 |
sed 's/ \{1,3\}//;s/\\$/&&/;H;s/.*//;x
     /\nS\nT\nR\nI\nN\nG\n1/s//STRING2/
     /\nS\nT\nR\nI\nN\nG\n2/s//STRING1/
     /\\n/!{x;d};s/\n//g;s/./\\&/g' |
     xargs printf %b

###OUTPUT###

STRING2STRING2

STRING1STRING2
STRING1

Something like this will always replace each occurrence of your target strings only once as they occur in sed's in stream at one bite per line. This is the fastest way I can imagine you'd do it. Then again, I don't write C. But this does reliably handle null delimiters if you wish it. See this answer for how it works. This has no problems with any contained special shell characters or similar - but it is ASCII locale specific, or, in other words, od will not output multi-byte characters on the same line and will only do one per. If this is a problem you'll want to add in iconv.

Answer 3

A perl solution. Even if some stated it is not possible, I found one but in general a simple match and replace is not possible and even it gets worse because of the backtracking of a NFA the result can be unexpected.

In general, and this must be said, the problem yelds different results which depend on the order and length of the replacement tuples. i.e.:

A B
AA CC

and the input AAA results in BBB or CCB.

Here the code:

#!/usr/bin/perl

$v='if (0) {} ';
while (($a,$b)=split /\s+/, <DATA>) {
  $k.=$a.'|';
  $v.='elsif ($& eq \''.$a.'\') {print \''.$b.'\'} ';
}
$k.='.';
$v.='else {print $&;}';

eval "
while (<>) {
  \$_ =~ s/($k)/{$v}/geco;
}";  
print "\n";


__DATA__
A    B
B    A
abba baab
baab abbc
abbc aaba

Checkerbunny:

$ echo 'ABBabbaBBbaabAAabbc'|perl script
$ BAAbaabAAabbcBBaaba