I have a file

$ cat f2
line 1; li
ne 2$

where note that the last $ is bash prompt, not part of the file content.

I try to concatenate each line which doesn't end in a digit with its next line with gawk. But unlike my previous post, now I try to figure out how backslashes are handled by bash, gawk and gensub(), by experimenting with different number of backslashes in front of the new line character \n. I was wondering why gawk commands with more than three backslashes before n fail to find a line not ending in a digit, and succeed when otherwise? Generally, how are backslashes processed succesively by bash, gawk and gensub()? Thanks.

$ gawk 'BEGIN{RS="\f"} {b=gensub("([^[:digit:] ]) *\n", "\\1", "g"); print b}' f2
line 1; line 2
$ gawk 'BEGIN{RS="\f"} {b=gensub("([^[:digit:] ]) *\\n", "\\1", "g"); print b}' f2
line 1; line 2
$ gawk 'BEGIN{RS="\f"} {b=gensub("([^[:digit:] ]) *\\\n", "\\1", "g"); print b}' f2
line 1; line 2
$ gawk 'BEGIN{RS="\f"} {b=gensub("([^[:digit:] ]) *\\\\n", "\\1", "g"); print b}' f2
line 1; li
ne 2

Can someone explain what gawk and gensub() see when \n, \\n, \\\n, and \\\\n pass through bash and gawk respectively?

Take \n as example, does bash not modify it (because of single quotes in bash) so gawk sees \n? Does gawk modify \n to be n so gensub() sees n, and if yes, why can gensub() know it is a newline to match?

  • 1
    Regarding the bash part, if you're using single quotes ('...') then everything including backspaces are preserved as is, so they're all passed down to gawk. – filbranden Nov 14 '18 at 4:38

In bash, '...' are strong quotes, so with '\n', a literal \n is passed to awk and with '\\n', a literal \\n. There's no transformation.

In awk, inside "...", \n and \\... are expanded. So when passed "\n" to gensub() (or print or anything in awk), that's an actual newline character, and when passed "\\", that's a \.

Now, gensub() also understands its first argument as a regular expression, where \ also has a special meaning which varies between implementations.

What is consistent between implementations is that a \\ regexp matches a literal \ just like \. matches a literal .. However for a \n regexp, whether that matches a newline character or a n varies with the implementation. In the case of gawk, that matches on newline. So both gensub("\n", "x") and gensub("\\n", "x") replace newline characters with x, the first one because a literal newline character is passed to gensub(), the second because \n is passed to gensub() which is understood as a regexp that matches a newline character.

Note that the POSIX specification used to have several issues when it came to backslash processing in regular expressions in awk. That will be corrected in the next version of the specification. See http://austingroupbugs.net/view.php?id=1105 for details.

It gets even more confused when using /\n/ instead of "\n".

  • Thanks. I seem to remember you have posted some shell commands for automatically double backslashes in a string (possibly for the same purpose). Do you happen to remember whether and where you have posted it? – Tim Nov 14 '18 at 16:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.