I have input looking like this:


I would like to read such input into an array in AWK, to then process it together with the embedded newlines. Is that possible? It would be better to not use GNU features.


With posix awk you can use getline

awk '{while(/\\$/){getline tmp;$0=$0"\n"tmp}print "<LINE>"$0"<LINE>"}' file

Just keeps adding the next line as long as the last line end in \.


Same in perl

perl -ne '$_.=<> while /\\$/;chomp;print "<LINE>$_<LINE>\n"' file
  • 1
    ... and same with sed: sed ':do;/\\$/{$!N;b do};s/.*/<LINE>&<LINE>/' infile – don_crissti May 26 '16 at 17:58

Using GNU awk:

$ awk '{printf "%s%s%s","line=",$0,RT}' RS='[^\\\\]\n' text

As you can see, the lines ending with \ are joined to the next line. This is because the record separator RS was redefined to be any non-backslash followed by a newline. In other words, backslash-newline is not a record separator.

A slight trick here is that the record separator swallows the last character of the record. That character, though, is saved for us in the builtin variable RT. A slight change to the program corrects the value of $0 at the beginning of the code so that that issue disappears:

$ awk '{$0=$0 substr(RT,1,1)} {print "line=",$0}' RS='[^\\\\]\n' text
line= entry1line1
line= entry2line1\
line= entry3line1

RT contains the whole of the actual observed record separator. In our case, that means that it has the last character of the record and the newline character which follows. Consequently, in the code above, substr is used to add the first character of RT to the end of $0.

  • It swallows the record separator, not the last character, the whole separator is stored in RT – 123 May 26 '16 at 8:49
  • @123 Yes, that is why substr was used in the second example. You point is good: I have added more text to the answer to explain. – John1024 May 26 '16 at 17:30

You can change the record-separator in POSIX awk by changing RS. POSIX does not specify whether that can be a regular expression, saying only

The unspecified behavior from using multi-character RS values is to allow possible future extensions based on extended regular expressions used for record separators. Historical implementations take the first character of the string and ignore the others.

However, you could either read the whole file as a string (by choosing an unlikely RS), or using getline, look at the ends of lines and splice the result together as needed.

  • Yes, but only to a single character (or none). How then to distinguish newline and backslash newline? – Michael Vehrs May 26 '16 at 8:14

The perl cookbook has an example of how to do this using perl.

I've adapted that example to work with <> (stdin and/or whatever filename(s) are given as args on the command line) rather than a named file-handle, and also to retain the newline after the \ continuation (which is slightly unusual - it's far more common to want a continued line to be treated as one long line, with continuations joined with either nothing or maybe a space character).

perl -e '
while (defined($line = <>) ) {
    if ($line =~ s/\\$//) {
        $line .= "\n" . <>;
        redo unless eof();
    # process full record in $line here
    printf "%04i:\"%s\"\n\n", $count++,$line;
}' willdavies.txt 

You can do whatever you want with $line after the comment that says # process full record.... I've chosen to just print each line as a separate paragraph with a zero-padded line counter. I've also added quotes around $line so you can see exactly what is in $line (and what is not).




  • If you chomp after appending the line, you don't have to re add the newline everytime. – 123 May 26 '16 at 9:26
  • That's true. However, leaving it as I wrote it makes it easy to change the script so that it joins the continued lines with a space or any other string (including nothing at all). – cas May 26 '16 at 9:28
  • Fair enough, just thought i'd point it out :) – 123 May 26 '16 at 9:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.