There must be a better way to replace single newlines only?

Question

I am in the habit of writing one line per sentence because I typically compile things to LaTeX, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the start of a new paragraph.

Now, I have a file written in this style which I'd like to just send as plain text. I want to remove all the single linebreaks but leave the double linebreaks intact. This is what I've done:

sed 's/^$/NEWLINE/' file.txt | awk '{printf "%s ",$0}' | sed 's/NEWLINE/\n\n/g' > linebreakfile.txt

This replaces empty lines with some text I am confident doesn't appear in the file: NEWLINE and then it gets rid of all the line breaks with awk (I found that trick on some website) and then it replaces the NEWLINEs with the requisite two linebreaks.

This seems like a long winded way to do a pretty simple thing. Is there a simpler way? Also, if there were a way to replace multiple spaces (which sometimes creep in for some reason) with single spaces, that would be good too.

I use emacs, so if there's some emacs specific trick that's good, but I'd rather see a pure sed or pure awk version.

Answer 1

You can use awk like this:

$ awk ' /^$/ { print; } /./ { printf("%s ", $0); } ' test

Or if you need an extra newline at the end:

$ awk ' /^$/ { print; } /./ { printf("%s ", $0); } END { print ""; } ' test

Or if you want to separate the paragraphs by a newline:

$ awk ' /^$/ { print "\n"; } /./ { printf("%s ", $0); } END { print ""; } ' test

These awk commands make use of actions that are guarded by patterns:

/regex/

or

END

A following action is only executed if the pattern matches the current line.

And the ^$. characters have special meaning in regular expressions, where ^ matches the beginning of line, $ the end and . an arbitrary character.

Answer 2

Use Awk or Perl's paragraph mode to process a file paragraph by paragraph, where paragraphs are separated by blank lines.

awk -vRS= '
  NR!=1 {print ""}      # print blank line before every record but the first
  {                     # do this for every record (i.e. paragraph):
    gsub(" *\n *"," "); # replace newlines by spaces, compressing spaces
    sub(" *$","");      # remove spaces at the end of the paragraph
    print
  }
'
perl -000 -pe '             # for every paragraph:
  print "\n" unless $.==1;  # print a blank line, except before the first paragraph
  s/ *\n *(?!$)/ /g;        # replace newlines by spaces, compressing spaces, but not at the end of the paragraph
  s/ *\n+\z/\n/             # normalize the last line end of the paragraph
'

Of course, since this doesn't parse the (La)TeX, it will horribly mutilate comments, verbatim environments and other special-syntax. You may want to look into DeTeX or other (La)TeX-to-text converters.

Answer 3

(reviving an ancient question)

This seems to be exactly what fmt and par are for - paragraph reformatting. Like you (and also like many programs) they define paragraph boundaries as one (or more) blank lines. Try piping your text through one of these.

fmt is a standard unix utility and can be found in GNU Coreutils.

par is a greatly-enhanced fmt written by Adam M. Costello which can be found at http://www.nicemice.net/par/ (it has also been packaged for several distributions, including debian - I packaged it for debian in Jan 1996, although there's a new maintainer for the pkg now.).

Answer 4

Sed Solution

$ sed -e ':a;N;$!ba;s/\(.\)\n/\1 /g' -e 's/\n/\n\n/' test.text

Note, that in this solution :a is creating a label and not using the a command.

Replacing Multiple Spaces

Use tr: $ tr -s ' ' <test.text

Answer 5

If I've understood correctly, an empty line implies two consecutive newlines, \n\n.

If so, one possible solution would be to eliminate all singular occurrences of newlines.

In Perl, a lookahead assertion is one way to achieve this:

$ perl -0777 -i -pe 's/\n(?=[^\n])//g' test

• The -0777 flag effectively slurps the whole file into a single string
• -p tells perl to print the string it's working on by default
• -i specifies in-place editing
• Global matching ensures that all single newline occurrences are dealt with

Answer 6

sed -e'/./{H;$!d;}' -e'x;s/\n//g'

sed will append any line to Hold space which contains at least a single character. It immediately thereater deletes all of those excepting perhaps the last. The only lines which can remain are blanks, and it is on these lines when sed exchanges the hold and pattern spaces and deletes all accumulated \newline characters.

If you want lines which contain only <tabs> or <spaces> to be considered blank, replace the /./ address above with /[^[:blank:]]/. To also squeeze spaces do:

 sed -e'/./{H;$!d;}'    \
     -e'x;s/\n//g'      \
     -e's/\([[:blank:]]\)*/\1/g'

Answer 7

This might be old school:

(echo ".pl 1" ; echo ".ll 80" ; echo ".ad l" ; cat your_file) | nroff

This will output your text left aligned (.ad l), with line length of 80 (.ll 80). The page length option (.pl) tells the text processor to do page padding for page length of 1, so no page padding.

If you want all your paragraphs on a single line, you could use a large number for .ll:

(echo ".pl 1" ; echo ".ll 1000000" ; echo ".ad l" ; cat your_file) | nroff

man 7 groff for more formatting options.

Answer 8

Here's yet another sed solution that concatenates all lines into sed's "hold space" so that we get one long string that finally gets copied to the "pattern space" for pattern matching.

As newlines will be preserved in the final long string in sed's "pattern space", empty lines in terms of double linebreaks [^\n]\n\n[^\n] can be matched and modified to [^\n]\n[^\n].

For more information see, for example, sed and Multi-Line Search and Replace.

text='
line 1

line 2
line 3





line 4


line     5



line 6
line 7

line 8
'

# FreeBSD sed
# first sed deletes first / last line if empty and squeezes multiple spaces
printf '%s' "$text" |
sed -e '1{/^$/d;}' -e '${/^$/d;}' -e '/[[:space:]]\{2,\}/s// /g' | 
sed -n -e '1h;1!H;${;g;/\([^[:cntrl:]]\)\n\n\([^[:cntrl:]]\)/s//\1\
\2/g;p;}' |
nl -b a


# GNU sed
# alternative using ...;x;... instead of ...;g;...
# cf. man sed | less -p '\]x'
printf '%s' "$text" |
gsed -e '1{/^$/d;}' -e '${/^$/d;}' -e '/[[:space:]]\{2,\}/s// /g' | 
gsed -E -n '1h;1!H;${;x;/([^\n])\n\n([^\n])/s//\1\
\2/g;p;}' | 
nl -b a


# remove all the single linebreaks but leave the double linebreaks intact
printf '%s' "$text" | 
   sed -n -e '1h;1!H;${;g;/\([^[:cntrl:]]\)\n\([^[:cntrl:]]\)/s//\1 \2/g;p;}' | 
   nl -b a

Answer 9

After seeing Gilles' perl and awk compact examples, I was reluctant to post this, but I had already gone through the exercise, and it is a functioning script, which is reasonably documented; this point alone may be of interest to some... (sed with comments! :) )

This script:

• Considers blank lines to be blank even if they contain whitespace.
• Multiple spaces in the text are condensed to a single space.
• Trailing whitespace is removed from the text lines.
• Consecutive blank lines are collapsed to a single line.
• Leaves top and bottom blank lines intact.

For anything more than the most trivial scripts, sed can be written much more easily in a structured form, as a separate script file. Here is such an example.

Using extended regex syntax, call:

$ sed -rf script text-file

where the script looks as follows:

:first-empty-line
#================
/^[[:space:]]*$/ { # if pattern-space is empty...
    $q  # last line # flush-quit 
    n   # pattern-flush=nextline-continue
     
    :subsequent-empty-line
    #=====================
    /^[[:space:]]*$/ { # if pattern-space is empty...
        $d        # last line # pattern-delete-cycle
        N         # pattern+=nl+nextline
        s/.*\n//  # scrap the leading 'blank' line
        t subsequent-empty-line # branch-on-substitute
    }
}

:text-line
#=========
$q                       # last line # flush-quit 
s/^(.*)[[:space:]]*/\1/  # trim trailing whitespace
s/ +/ /g                 # condense mulltiple spaces
N                        # pattern+=nl+nextline
/^.*\n[[:space:]]*$/ { # if newly-read line is blank 
    P          # pattern-first-line-print
    s/^.*\n//  # remove the leading 'text' line
    t first-empty-line   # branch-on-substitute
}
# read line is text
s/\n/ /      # replace \n with a space
t text-line  # branch-on-substitute

Note: flush, in the comments, means: send the pattern-space to sed's internal stdout handling. It does not mean a definite print to stdout. The output is dependent on sed's -n option. eg. the q command means flush and quit... Compare these two snippets: echo x |sed -e q prints x, echo x |sed -ne q prints nothing, whereas using the p command would print 'x' twice or once, depending on the -n option.

Answer 10

In Emacs, I sometimes use this regex:

^J\([^^J]\) -> \1

Means:

replace every newline that is followed by something which is NOT a newline with only the thing, that followed the newline That way I get rid of all newlines within a paragraph but keep paragraphs (double-newlines)

Answer 11

The most compact (but simplistic) solution is:

awk -v RS='' '{gsub(/\n/," ")}1' file

It doesn't detect comments, or quoted strings inside the text, or several other higher level constructs. But it is very simple and converts this file:

$ cat file
one line.
second line.
join all the lines.
But ignore #comments?
but keep an empty line as empty.
As it is a paragraph delimiter.

Second paragraph
In this lines
should be after an empty line


Or, maybe, it should not.
And we will get only joined lines.

Into this (maybe better) file:

$ awk -v RS='' '{gsub(/\n/," ")}1' file

one line. second line. join all the lines. But ignore #comments? but keep an empty line as empty. As it is a paragraph delimiter.
Second paragraph In this lines should be after an empty line
Or, maybe, it should not. And we will get only joined lines.

Simple, fast, not perfect. But let's not make perfect the enemy of good.

How it works.

It works by setting RS to the empty string (NULL). That makes awk to work in paragraph mode:

Records
Normally, records are separated by newline characters.
If RS is set to the null string, then records are separated by empty lines.

Where "empty lines" are detected by two consecutive newlines (\n).

After that split happens, the $0 contains the whole paragraph (including single newlines). So, replacing new lines with an space ends joining all lines that do not have an empty line in between.

Answer 12

Using Raku (formerly known as Perl_6)

raku -e '.put for slurp.subst(:global, / \N <(\n)> \N /, " ");' 

OR

raku -e '.put for lines.join("\n").subst(:global, / \N <(\n)> \N /, " ");' 

The two lines of code above are identical except first example slurps in the whole file at once, while the second example lazily reads in lines and (since lines auto-chomps input) joins them back together with \n newlines. In both examples then a substitution is performed wherein \n newlines surrounded by \N non-newline characters are converted to single blank spaces (<(…)> denotes capture markers).

The OP also asks for code to regularize multiple blank lines between "paragraphs". At the end of either code example above, a second substitution can be added which is .subst(/ \n**3..* /, "\n\n", :global);. This takes occurrences of \n**3..* two-or-more consecutive blank lines and regularizes them to \n\n a single blank line. Be sure to join the two subst method calls with a . dot.

Sample Input:

One sentence.
Second sentence.
Third sentence; end of paragraph 1.

One sentence.
Second sentence; end of paragraph 2.


New section following two blank lines, first sentence.
Second sentence.
Third sentence.
Fourth sentence; end of new section a.k.a. paragraph 3.

Sample Output (using both subst calls described above):

One sentence. Second sentence. Third sentence; end of paragraph 1.

One sentence. Second sentence; end of paragraph 2.

New section following two blank lines, first sentence. Second sentence. Third sentence. Fourth sentence; end of new section a.k.a. paragraph 3.

Finally, I feel Raku does a really good job offering alternatives to the examples above, such as using the split routine to regularize \n newlines. If all you want to do is compact paragraphs then use the simple code below to store your text:

raku -e '.raku.put for slurp.split(/ \n**2..* /, :skip-empty);'  

Output:

"One sentence.\nSecond sentence.\nThird sentence; end of paragraph 1."
"One sentence.\nSecond sentence; end of paragraph 2."
"New section following two blank lines, first sentence.\nSecond sentence.\nThird sentence.\nFourth sentence; end of new section a.k.a. paragraph 3."

You can then restore individual "paragraphs" by prepending with put , terminating the line with ; semicolon, and running Raku again to restore each sentence back onto its own line.

https://raku.org

Answer 13

It turns out that with auto-fill-mode on, emacs does a pretty good job for my simple use cases with just M-q...