sed replace newline character with space

Question

How can I use sed to replace new line character with any other character?

Input:

 I cannot conceive that anybody will    
 require multiplications at the rate of 
 40,000 or even 4,000 per hour ...      

 -- F. H. Wales (1936)                  

Desired output:

I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ...  -- F. H. Wales (1936)

I've tried:

> pbpaste | sed 's/\n/ /g' 

but it outputs the same thing as input. I know it's a newline char because I've checked it with cat -ev and it prints $ as expected.

What else would be a good command to do this?

---

This shows where there is extra space between new line. I want to remove that as well. So it's like a sentence with spaces.

> pbpaste | cat -ev
 I cannot conceive that anybody will    $
 require multiplications at the rate of $
 40,000 or even 4,000 per hour ...      $
                                        $
 -- F. H. Wales (1936)                  ⏎   

Answer 1

tr is probably a better tool for this job. Try the following

pbpaste | tr '\n' ' '

With your input, I get the following output.

I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ...  -- F. H. Wales (1936) 

Answer 2

You can do it with sed (even without the GNU -z slurp extension) by using the N command in a loop:

$ sed -e :a -e '$!N;s/ *\n */ /;ta' -e 'P;D' input
 I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ... -- F. H. Wales (1936)                  

For discussion of an essentially equivalent command see for example 40. Append a line to the previous if it starts with an equal sign "=" in Sed One-Liners Explained, Part I

Answer 3

I agree with @QIS.

There are a couple of reasons why sed won't work for you.

One is that, by default, sed processes its input a line at a time, and for that reason, sed will never see a newline as part of a line. It's a delimiter between lines, by default.

Another is that, although GNU sed implements the '\n' representation of a newline, BSD sed (the version found on OSX) does not.

Just for any future GNU sed users who may read your question, GNU sed can be told to treat the input file as a NUL-delimited list of strings using the -z option. Assuming that your input contains no NUL bytes, this could be an option for GNU sed users:

$ pbpaste | sed -z 's/\n/ /g'

One trivial (and non-recommended) method would be to take advantage of the fact that when echo receives unquoted newlines as a result of command substitution, it replaces them with spaces. Also, echo will convert multiple spaces (or newlines) to a single space. Thus, it is likely that this will work for you also:

echo $(pbpaste)

Answer 4

Both fmt and par are good tools for re-formatting text.

Both of them can be used to reformat your text into a single long line after the linefeeds have been replaced with spaces (e.g. with tr):

$ tr '\n' ' ' < input.txt | fmt -w 999
I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ...    -- F. H. Wales (1936)

$ tr '\n' ' ' < input.txt | par w999
I cannot conceive that anybody will require multiplications at the rate of 40,000 or even 4,000 per hour ... -- F. H. Wales (1936)

Note that the output of fmt and par is slightly different. fmt has four spaces between the ... and the --, while par reduces them to one space.

---

More on fmt and par:

fmt is a standard utility that has been around for decades, and should be available on almost any unix system. On GNU/Linux systems, it's in the GNU coreutils package.

$ fmt < input.txt 
I cannot conceive that anybody will require multiplications at the rate
of 40,000 or even 4,000 per hour ...

  -- F. H. Wales (1936)

It is, however, very simple and does not allow much control over how the paragraphs can be reformatted, nor does it perform any special handling of leading and/or trailing characters (like > quoting in email or /* ... */ comments), which can lead to a jumbled mess.

par is much more flexible and capable. It can reformat C style /* ... */ comments and boxed text, email messages with multiple levels of quoting, and more. The following example doesn't show what it's really capable of.

$ par < input.txt 
I cannot conceive that anybody will require multiplications at the rate of
40,000 or even 4,000 per hour ...

  -- F. H. Wales (1936)

The following example (from man par) begins to show par's capabilities:

Before:

   John writes:
   : Mary writes:
   : + Anastasia writes:
   : + > Hi all!
   : + Hi Ana!
   : Hi Ana & Mary!
   Please unsubscribe me from alt.hello.

After "par Q+:+ q":

   John writes:

   : Mary writes:
   :
   : + Anastasia writes:
   : +
   : + > Hi all!
   : +
   : + Hi Ana!
   :
   : Hi Ana & Mary!

   Please unsubscribe me from alt.hello.

I've been using it daily from within vi/vim since I discovered the 1990s to reformat emails that I'm writing (or emails from others with unreadable long lines), code comments and other text. IMO, par is an indispensible, must-have program.