4

I'm trying to join lines with POSIX sed.

With GNU sed (without --posix) this works as intended:

$ sed ':a; N; s/\n//; b a' <<< $'a\nb\nc'
abc

But if I use --posix I get no output.

Why is this and how can I how can I do it otherwise?

2
  • A POSIX sed prints sed: Label too long: :a; N; s/\n//; b a
    – schily
    Commented Feb 22, 2020 at 22:12
  • Is the data always read from standard input, or could it be lines in a file on disk?
    – Kusalananda
    Commented Feb 23, 2020 at 15:12

2 Answers 2

5

That's a job for paste:

printf '%s\n' a b c | paste -sd '\0' -

(no, that's not joining with NULs, that's joining with no separator as required by POSIX. Some paste implementations also support paste -sd '' - but that's neither standard nor portable).

Note that except in the busybox implementation, it produces one empty line as output if the input is empty (a historical bug/misfeature unfortunately now engraved in the POSIX specification).

With POSIX sed:

sed -e :a -e '$!{N;ba' -e '}' -e 's/\n//g'

Or:

sed '
:a
$!{
  N
  ba
}
s/\n//g'

The b, :, and } commands cannot be followed by another command. In earlier versions of the POSIX specification, b a;s/a/b/ would require b to branch to the label called a;s/a/b/, in newer versions of the specification, it's now unspecified, to allow the GNU sed behaviour. The following command has to be in a subsequent expression or on a separate line.

Also POSIX requires N on the last line to exit without printing the pattern space. GNU sed only does it in POSIX mode, like when there's a POSIXLY_CORRECT variable in the environment or with your --posix option which explains why you get no output with --posix.

Also note that the minimum size of the pattern space guaranteed by POSIX is 8192 bytes. You can only use that approach portably for very small files. paste has no size limitation, and contrary to the sed approach doesn't need to load the whole file in memory before printing it.

Another approach is tr -d '\n'. However note that contrary to paste/sed, it produces a non-delimited line on output (outputs abc instead of abc\n on the example above).

In any case, <<< is a zsh operator (now supported by a few other shells), and $'...' is a ksh93 operator (now supported by most other POSIX-like shells), neither are POSIX sh operators (though the latter is likely to be added to the next major revision of the standard), so should not be used in POSIX sh scripts.

2
  • 1
    @EdMorton, you're right (except for the busybox implementation of paste which is not POSIX compliant in this instance)! I hadn't realised that. I'll edit it in. Commented Feb 23, 2020 at 15:36
  • A sed 'H;1h;$!d;g;s/\n//g' is also POSIX, has no branches (and thus no labels), so it needs no additional -e sections (easier to type).
    – user232326
    Commented Feb 24, 2020 at 17:48
3

With any awk in any shell on every UNIX box for any size input file (unless that input file is just 1 massive line of text containing no spaces that can't fit in memory):

$ printf '%s\n' a b c | awk -v ORS= '1; END{if (NR) print RS}'
abc
3
  • An awk '{ printf( "%s", $0 ) } END { if (NR) print "" }' doesn't need to change AWK internal variables.
    – user232326
    Commented Feb 28, 2020 at 22:03
  • Right, either approach is fine. I'd guess that setting ORS to null and letting a normal print happen might be a bit more efficient than calling printf but it'll be a drop in the ocean. Btw they don't hurt but you don't need parens around the printf arguments as it's a builtin language construct, not a function.
    – Ed Morton
    Commented Feb 28, 2020 at 22:14
  • You seem to agree, then talk about speed. (1) Coding require choices. The printf option is more flexible in the conditions it could be used, has less requirements. (2)Yes, the ORS= is faster, twice faster. (3) But, if you do need speed, you may want to try some other utility and walking out of AWK, try tr -d '\n' (and code for a newline). It is about 10 times faster. Is that too far away from your "zone of comfort"?. (4) If the parens are allowed (and indeed they are), then, it is matter of preference. I prefer to clearly mark start and end. (5) Why do you get confused with a function?
    – user232326
    Commented Feb 29, 2020 at 15:18

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .