1

I have a string that contains newline characters. I want to escape all newlines in that string by replacing all newline characters with a string of two characters: "\n". How can I do this in POSIX sh?

Here's the goal:

$ printf 'a\nb\nc\nd' | escape_newlines | od -a
0000000   a   \   n   b   \   n   c   \   n   d
        141 134 156 142 134 156 143 134 156 144
0000012

How do I define escape_newlines?

Methods I have tried:

  • tr — Problem: not able to convert single character into multiple characters.

  • awk 'BEGIN{ORS="\\n"} {print}' — Problem: always inserts the two-character string "\n" at the end of the string even if the string does not end with a newline character. Example:

    $ printf 'hi\n' | awk 'BEGIN{ORS="\\n"} {print}' | od -ab
    0000000   h   i   \   n
            150 151 134 156
    0000004
    $ printf 'hi' | awk 'BEGIN{ORS="\\n"} {print}' | od -ab
    0000000   h   i   \   n
            150 151 134 156
    0000004
    
  • sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g' — Problem: if there is a newline character at the end of the string, it will not be converted. Example:

    $ printf 'h\ni' | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g' | od -ab
    0000000   h   \   n   i
            150 134 156 151
    0000004
    $ printf 'h\ni\n' | sed -e ':a' -e 'N' -e '$!ba' -e 's/\n/\\n/g' | od -ab
    0000000   h   \   n   i  nl
            150 134 156 151 012
    0000005
    
0

1 Answer 1

1

Try awk with:

string='x
y
'
new_string=$(
  LC_ALL=C awk -- '
    BEGIN {
      gsub("\n", "\\n", ARGV[1])
      printf "%s", ARGV[1]
    }' "$string"
)

In any case, note that command substitution removes all trailing newline characters. OK here as the output of awk doesn't contain any, but that means we could also have used print instead of printf "%s".

With sed:

new_string=$(
  printf '%s\n' "$string" |
    LC_ALL=C sed '
      :1
      $ ! {
        N
        b1
      }
      s/\n/\\n/g'
)

Note that per POSIX, using N on the last line is meant to discard the pattern space and exit. GNU sed only does it when $POSIXLY_CORRECT is in the environment, but still exits upon N called on the last line (but still prints the pattern space).

We use LC_ALL=C to avoid potential issues with decoding the string in the charmap of the user's locale.

sed is a text utility, so it expects text input and produces text output. Something that is not empty and doesn't end in a newline character is not text. Here we add one newline to the input, and rely on command substitution to remove the one sed adds on output.

Also note that if the input has lines with a length in bytes larger than LINE_MAX (which can be as low as 1024), that makes it non-text as well, and the behaviour is unspecified. IIRC, the pattern space is not required to be able to hold more than 10 x LINE_MAX as well.

The awk approach will have some limits as well starting with ARG_MAX which on systems will be lower than 10 x LINE_MAX. That limit applies to the sed one as well with shells where printf is not builtin (such as ksh88 or pdksh-based ones).

There's no limit to the size of a shell variable, though if it's exported to the environment, it will run against the ARG_MAX limit for all the external commands that are executed.

To process a stream, you'd need something like:

... | (cat; echo) | LC_ALL=C awk '
  {printf "%s", sep $0; sep = "\\n"}'

Though beware that output is not text so cannot be processed by a POSIX text utility.

2
  • How do I use the awk solution in a pipeline as shown in the escape_newlines example goal in the question?
    – Flux
    Nov 20, 2023 at 10:41
  • @Flux, see edit. Nov 20, 2023 at 11:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .