Escape characters in sed transliterate

Question

I want to use sed's transliterate (y///) to replace one set of characters by another.

I would expect this to work just as well as using the tr program.

$ echo '[]{}abc' | tr '[ab}' 'gefh' 
g]{hefc

However, when I go to perform this same operation with sed, I see the following error:

$ echo '[]{}abc' | sed 'y/[ab}/gefh/' 
sed: 1: "y/[ab}/gefh/": unbalanced brackets ([])

This makes some sense, as I expect to need to escape the [ character. However, when I do try and escape that, I receive the following, different error:

$ echo '[]{}abc' | sed 'y/\[ab}/gefh/' 
sed: 1: "y/\[ab}/gefh/": transform strings are not the same length

My current work-around is to either (1) just use tr or (2) insert a "dummy character" in the right-hand side of the transliteration whose job is to do nothing but match the escape character.

$ echo '[]{}abc' | sed 'y/\[ab}/_gefh/' 
g]{hefc

This is however unsatisfying and suspicious. It's also not very safe, e.g. when \ is in the input string.

$ echo '[]{}abc\' | sed 'y/\[ab}/_gefh/' 
g]{hefc_

What's the correct way to escape a character in a sed transliteration without sed treating the escape character itself a part of the translation?

Answer 1

Assuming you are on macOS (the only system whose native sed I can get to display this issue, although I haven't checked on FreeBSD whence macOS's sed comes):

$ echo '[]{}abc' | sed 'y/[ab}/gefh/'
sed: 1: "y/[ab}/gefh/": unbalanced brackets ([])

$ echo '[]{}abc' | sed 'y/\[ab}/gefh/'
sed: 1: "y/\[ab}/gefh/": transform strings are not the same length

$ echo '[]{}abc' | sed 'y/\[ab}/\gefh/'
g]{hefc

So, one solution is to

1. Escape the [ in the first string to avoid having an unbalanced bracket, and
2. Make the two strings equal length by adding a "no-op" backslash to the second string too.

Or,

• You could also enclose both strings in [...], which, upon reflection, may be the safest way to deal with this as it can be done in a mechanical manner without caring about where in the string the [ is located:

  $ echo '[]{}abc' | sed 'y/[[ab}]/[gefh]/'
  g]{hefc
  

• Or install GNU sed via e.g. Homebrew on macOS, or FreeBSD's package system, and use that.

I would treat this as a bug in this sed implementation.

Answer 2

What you're doing is the correct way: [ is always supposed to be an ordinary character in sed y///. This is different from tr where [ can be part of a character class like [:alpha:].

Unfortunately, several implementations of sed evidently have a bug that causes them to try to parse balanced brackets in sed. I've observed the bug you describe with FreeBSD 11.2 and with BusyBox 1.30.1.

Using backslashes is tricky: the behavior of backslash+character is not standard except when the character is \, n or the delimiter. So while you can perhaps use it to work around the bug in your specific implementation, the resulting code probably won't work with another implementation.

A portable workaround is to temporarily swap [ with some other character that you don't want to change, and in the same replacement translate ] to itself. To avoid likely possible buggy parses, this character shouldn't be ], ^ or :. When you're doing the swap, you'll need to construct a string with balanced brackets and with something between them. FreeBSD doesn't like [] in the replacement either; an easy way to work around this is to put an extra character before the ]. For example, I'll use B as the temporary [.

y/[B_]/B[_]/; y/Bab}/gefh/; y/[B_]/B[_]/