Replace non-printable characters in perl and sed

Question

I need to replace some non-printable characters with spaces in file.

Specifically, all characters from 0x00 up to 0x1F, except 0x09 (TAB), 0x0A (new line), 0x0D (CR)

Up until now, I just needed to replace 0x00 character. Since my previous OS was AIX (without GNU commands), I can't use sed (well, I can but it had some limitations). So, I found next command using perl, which worked as expected:

perl -p -e 's/\x0/ /g' $FILE_IN > $FILE_OUT 

Now I'm working on Linux, so I expected to be able to use sed command.

My questions:

• Is this command appropriate to replace those characters? I tried, and it seems to work, but I want to make sure:

  perl -p -e 's/[\x00-\x08\x0B\x0C\x0E-\x1F]/ /g' $FILE_IN > $FILE_OUT  
  

• I thought perl -p works as sed. So, why does the previous command work (at least, it doesn't fail), and next one doesn't?

  sed -e 's/[\x00-\x08\x0B\x0C\x0E-\x1F]/ /g' $FILE_IN > $FILE_OUT   
  

  It tells me:

  sed: -e expression #1, char 34: Invalid collation character

Answer 1

That's a typical job for tr:

LC_ALL=C tr '\0-\10\13\14\16-\37' '[ *]' < in > out

In your case, it doesn't work with sed because you're in a locale where those ranges don't make sense. If you want to work with byte values as opposed to characters and where the order is based on the numerical value of those bytes, your best bet is to use the C locale. Your code would have worked with LC_ALL=C with GNU sed, but using sed (let alone perl) is a bit overkill here (and those \xXX are not portable across sed implementations while this tr approach is POSIX).

You can also trust your locale's idea of what printable characters are with:

tr -c '[:print:]\t\r\n' '[ *]'

But with GNU tr (as typically found on Linux-based systems), that only works in locales where characters are single-byte (so typically, not UTF-8).

In the C locale, that would also exclude DEL (0x7f) and all byte values above (not in ASCII).

In UTF-8 locales, you could use GNU sed which doesn't have the problem GNU tr has:

sed 's/[^[:print:]\r\t]/ /g' < in > out

(note that those \r, \t are not standard, and GNU sed won't recognize them if POSIXLY_CORRECT is in the environment (will treat them as backslash, r and t being part of the set as POSIX requires)).

It would not convert bytes that don't form valid characters if any though.

Answer 2

I was trying to send a notification via libnotify, with content that may contain unprintable characters. The existing solutions did not quite work for me (using a whitelist of characters using tr works, but strips any multi-byte characters).

Here is what worked, while passing the 💩 test:

message=$(iconv --from-code=UTF-8 -c <<< "$message")