Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I am in the habit of writing one line per sentence because I typically compile things to LaTex, or am writing in some other format where line breaks get ignored. I use a blank line to indicate the start of a new paragraph.

Now, I have a file written in this style which I'd like to just send as plain text. I want to remove all the single linebreaks but leave the double linebreaks intact. This is what I've done:

sed 's/$^/NEWLINE/' file.txt | awk '{printf "%s ",$0}' | sed 's/NEWLINE/\n\n/g' > linebreakfile.txt

This replaces empty lines with some text I am confident doesn't appear in the file: NEWLINE and then it gets rid of all the line breaks with awk (I found that trick on some website) and then it replaces the NEWLINEs with the requisite two linebreaks.

This seems like a long winded way to do a pretty simple thing. Is there a simpler way? Also, if there were a way to replace multiple spaces (which sometimes creep in for some reason) with single spaces, that would be good too.

I use emacs, so if there's some emacs specific trick that's good, but I'd rather see a pure sed or pure awk version.

share|improve this question
    
You meant ^$, not $^ in the first sed-command. –  user unknown Feb 5 '11 at 10:27
    
@user yes, yes I did. –  Seamus Feb 5 '11 at 15:01
    
An easier way to remove all the line breaks: tr -d "\n". –  jfgagne Aug 11 '11 at 18:16

10 Answers 10

up vote 10 down vote accepted

You can use awk like this:

$ awk ' /^$/ { print; } /./ { printf("%s ", $0); }' test

Or if you need an extra newline at the end:

$ awk ' /^$/ { print; } /./ { printf("%s ", $0); } END { print; } ' test

Or if you want to separate the paragraphs by a newline:

$ awk ' /^$/ { print "\n"; } /./ { printf("%s ", $0); } END { print; } ' test

These awk commands make use of actions that are guarded by patterns:

/regex/

or

END

A following action is only executed if the pattern matches the current line.

And the ^$. characters have special meaning in regular expressions, where ^ matches the beginning of line, $ the end and . an arbitrary character.

share|improve this answer
    
This is good, although I'd prefer to keep the empty line between paragraphs. I assume you could do something like this by adding an extra new line somewhere in the first print command? Also, what is /./ doing: it seems to be acting like and else for the /^$/ string match, is that right? –  Seamus Feb 4 '11 at 21:26
1  
@Seamus, sure - just replace the first print (updated the answer) - /./ matches all lines which are at least one character long, i.e. the complement of the /^$/ pattern which matches only empty lines. –  maxschlepzig Feb 5 '11 at 10:39

Sed Solution

$ sed -e ':a;N;$!ba;s/\(.\)\n/\1 /g' -e 's/\n/\n\n/' test.text

Note, that in this solution :a is creating a label and not using the a command.

Replacing Multiple Spaces

Use tr: $ tr -s ' ' <test.text

share|improve this answer

If I've understood correctly, an empty line implies two consecutive newlines, \n\n.

If so, one possible solution would be to eliminate all singular occurrences of newlines.

In Perl, a lookahead assertion is one way to achieve this:

$ perl -0777 -i -pe 's/\n(?=[^\n])//g' test
  • The -0777 flag effectively slurps the whole file into a single string
  • -p tells perl to print the string it's working on by default
  • -i specifies in-place editing
  • Global matching ensures that all single newline occurrences are dealt with
share|improve this answer
    
One problem this has is that there are no spaces between the sentences. –  Steven D Feb 4 '11 at 22:10

Use Awk or Perl's paragraph mode to process a file paragraph by paragraph, where paragraphs are separated by blank lines.

awk -vRS= '
  NR!=1 {print ""}      # print blank line before every record but the first
  {                     # do this for every record (i.e. paragraph):
    gsub(" *\n *"," "); # replace newlines by spaces, compressing spaces
    sub(" *$","");      # remove spaces at the end of the paragraph
    print
  }
'
perl -000 -pe '             # for every paragraph:
  print "\n" unless $.==1;  # print a blank line, except before the first paragraph
  s/ *\n *(?!$)/ /g;        # replace newlines by spaces, compressing spaces, but not at the end of the paragraph
  s/ *\n+\z/\n/             # normalize the last line end of the paragraph
'

Of course, since this doesn't parse the (La)TeX, it will horribly mutilate comments, verbatim environments and other special-syntax. You may want to look into DeTeX or other (La)TeX-to-text converters.

share|improve this answer

It turns out that with auto-fill-mode on, emacs does a pretty good job for my simple use cases with just M-q...

share|improve this answer
    
The details of what auto-fill-mode does depend on what major mode you have active. –  dmckee Aug 12 '11 at 20:14

This might be old school:

(echo ".pl 1" ; echo ".ll 80" ; echo ".ad l" ; cat your_file) | nroff

This will output your text left aligned (.ad l), with line length of 80 (.ll 80). The page length option (.pl) tells the text processor to do page padding for page length of 1, so no page padding.

If you want all your paragraphs on a single line, you could use a large number for .ll:

(echo ".pl 1" ; echo ".ll 1000000" ; echo ".ad l" ; cat your_file) | nroff

man 7 groff for more formatting options.

share|improve this answer

Here's yet another sed solution that concatenates all lines into sed's "hold space" so that we get one long string that finally gets copied to the "pattern space" for pattern matching.

As newlines will be preserved in the final long string in sed's "pattern space", empty lines in terms of double linebreaks [^\n]\n\n[^\n] can be matched and modified to [^\n]\n[^\n].

For more information see, for example, sed and Multi-Line Search and Replace.

text='
line 1

line 2
line 3





line 4


line     5



line 6
line 7

line 8
'

# FreeBSD sed
# first sed deletes first / last line if empty and squeezes multiple spaces
printf '%s' "$text" |
sed -e '1{/^$/d;}' -e '${/^$/d;}' -e '/[[:space:]]\{2,\}/s// /g' | 
sed -n -e '1h;1!H;${;g;/\([^[:cntrl:]]\)\n\n\([^[:cntrl:]]\)/s//\1\
\2/g;p;}' |
nl -b a


# GNU sed
# alternative using ...;x;... instead of ...;g;...
# cf. man sed | less -p '\]x'
printf '%s' "$text" |
gsed -e '1{/^$/d;}' -e '${/^$/d;}' -e '/[[:space:]]\{2,\}/s// /g' | 
gsed -E -n '1h;1!H;${;x;/([^\n])\n\n([^\n])/s//\1\
\2/g;p;}' | 
nl -b a


# remove all the single linebreaks but leave the double linebreaks intact
printf '%s' "$text" | 
   sed -n -e '1h;1!H;${;g;/\([^[:cntrl:]]\)\n\([^[:cntrl:]]\)/s//\1 \2/g;p;}' | 
   nl -b a
share|improve this answer

After seeing Gilles' perl and awk compact examples, I was reluctant to post this, but I had already gone through the exercise, and it is a functioning script, which is reasonabley documented; this point alone may be of interest to some.. (sed with comments! :)

This script considers blank lines to be blank even if they contain whitespace.
Multiple spaces in the text are condensed to a single space.
Trailing whitespace is removed from the text lines. Consecutive blank lines are collapsed to a single line. The script leaves top and bottom blank lines intact.

For anything more than the most trivial scripts, sed can be written much more easily in a structured form, as a seperate script file. Here is such an example.

using extended regex syntax
call:$ sed -rf script text-file

  :first-empty-line
  #================
  /^[[:space:]]*$/ { # if pattern-space is empty...
      $q  # last line # flush-quit 
      n   # pattern-flush=nextline-continue

      :subsequent-empty-line
      #=====================
      /^[[:space:]]*$/ { # if pattern-space is empty...
          $d        # last line # pattern-delete-cycle
          N         # pattern+=nl+nextline
          s/.*\n//  # scrap the leading 'blank' line
          t subsequent-empty-line # branch-on-substitute
      }
  }

  :text-line
  #=========
  $q                       # last line # flush-quit 
  s/^(.*)[[:space:]]*/\1/  # trim trailing whitespace
  s/ +/ /g                 # condense mulltiple spaces
  N                        # pattern+=nl+nextline
  /^.*\n[[:space:]]*$/ { # if newly-read line is blank 
      P          # pattern-first-line-print
      s/^.*\n//  # remove the leading 'text' line
      t first-empty-line   # branch-on-substitute
  }
  # read line is text
  s/\n/ /      # replace \n with a space
  t text-line  # branch-on-substitute

Note: flush, in the comments, means: send the pattern-space to sed's internal stdout handling. It does not mean a definite print to stdout. The output is dependent on sed's -n option. eg. the q command means flush and quit... Compare these two snippets: echo x |sed -e q prints x, echo x |sed -ne q prints nothing, whereas using the p command would print 'x' twice or once, depending on the -n option.

share|improve this answer
    
+1 for good comments. I've seen too many programs with no comments at all. –  David Cary May 9 '13 at 15:17

(reviving an ancient question)

This seems to be exactly what fmt and par are for - paragraph reformatting. Like you (and also like many programs) they define paragraph boundaries as one (or more) blank lines. Try piping your text through one of these.

fmt is a standard unix utility and can be found in GNU Coreutils.

par is a greatly-enhanced fmt written by Adam M. Costello which can be found at http://www.nicemice.net/par/ (it has also been packaged for several distributions, including debian - I packaged it for debian in Jan 1996, although there's a new maintainer for the pkg now.).

share|improve this answer
sed -e '/./{H;$!d;}' -e 'x;P;y/\n/ /'
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.