3

I have a bunch of text files in the following format:

Lorem ipsum dolor sit amet,
consetetur sadipscing elitr,
sed diam nonumy eirmod tempor
invidunt ut labore et dolore
magna aliquyam erat, sed diam
voluptua. - At vero eos et accu-
sam et justo duo dolores et ea
rebum. - Stet clita kasd guber-
gren, no sea takimata sanctus
est Lorem ipsum dolor sit amet.

How can I print this as continuous text on the command line, but with removing the syllable division on the line ends:

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

I could use tr '\n' ' ' to convert the new-lines into spaces

The problem is tr can only replace one character and I would need some command to remove the -\n in advance. How can I achieve this on the bash comman-line?

2
  • 1
    Note that there may be cases where you want to keep the hyphens (e.g. in “a back-of-\nthe-envelope calculation”).  There may be no practical way to programmatically distinguish those cases from the others. – gidds May 4 at 0:17
  • @gidds you are 100% right. There is also situations where you want to keep the paragraph. Though the answers to my question did help me a lot to make life easier ;-) – nath May 4 at 11:22
5

Using awk:

awk -F'-$' '{ printf "%s", sep $1; sep=/-$/?"":OFS } END{ print "" }' infile

with the -F'-$', we defined the Field Separator to single hyphen at the end of line, so with this and by taking the first field $1, we will always have the line without that hyphen for those line having this hyphen or still entire line for those not having that hyphen.

then we do simply printing it with a sep in between but that changes when reading the next line to empty-string if current line was ending with a hyphen otherwise to OFS (Output Field Separator, default is Space character).

at the END{...} block we are adding a final newline character to make it a POSIX text file, if you don't want that to be added, just remove that part.


Using sed, alternatively:

sed ':loop /-$/N;s/-\n//;t loop; N;s/\n/ /;t loop' infile
  • :loop
    • if a line ended with a hyphen (testing with /-$/), do read the Next line and replace the "hyphen+\newline" with an empty string.
      if substitution was successful (testing with t), then jump to the label loop and process the next line and skip executing rest of the code.
    • else, read the Next line and replace the embedded \newline in between those two lines with a space character.
      if substitution here was also successful, then jump to the label loop and process the next line.
0
3
  1. Remove dashes used for hyphenation at the end of lines (joining hyphenated lines with the next line in the process).
  2. Replace newlines with spaces.
sed -e ':again' -e '/[[:alpha:]]-$/ { N; s/-\n//; b again; }' file |
paste -s -d ' ' -

This uses sed join hyphenated lines. It does this by detecting lines that have dashes that are immediately preceded by a letter at the end of lines. By matching [[:alpha:]] in front of the dash, I'm more certain that I'm only removing dashes used for hyphenation. The hyphenated lines are then joined with the next line by reading in the next line with N (which inserts a literal newline character) and deleting the -\n. The joined line is then re-checked for further hyphenation (this is what the b again branch to the again label does).

The paste utility is then used to paste all lines into one continuous line. The -s option is used to concatenate all lines into a single line, and -d ' ' is used to specify the delimiter character used in the concatenation of the lines.

I could have used tr '\n' ' ' in place of the paste command that I ended up with, but that would have given me an unterminated line (no newline character at the end).

0
3

Another perl solution (assuming the last line doesn't end with - character):

perl -pe 's/-\n//; s/\n/ / if !eof'

Unlike sed or awk, the record separator can be directly manipulated using perl. So, it is easy to delete if -\n is matched. s/\n/ / if !eof will then replace remaining newlines with space, except for the last line.

If you don't care about an extra space at the end, you can remove the if !eof part.

1
  • 1
    Could also use perl -0777 -pe 's/-\n//g; s/\n/ /g; s/ $/\n/' without worrying about the last line. – glenn jackman May 3 at 17:17
2

Using the POSIX sed stream editor utility.

sed -e ':a
  $!N
  s/-\n//;ta
  s/\n/ /;ta
' file

With awk utility, we setup a loop to read the next line and manipulate the ORS for printing the previous line.

awk '
{
  while (getline t > 0) {
    ORS = sub(/-$/, "") ? "" : OFS
    print
    $0 = t
  }
  ORS = RS
}
1' file

We can rearrange the output record separator $\ based on the condition of the current record.

perl -lpe '
  $\ =   eof  ? $/
     : s/-$// ? $,
     :          $"
     ;
' file
  • $/ is the input record separator and defaults to a newline.
  • $" is the array element separator under quotes and defaults to a space.
  • $, is the output field separator and defaults to an empty string.
2

Depending on how you'd want blank lines handled this:

$ awk -v RS= '{gsub(/-\n/,""); $1=$1} 1' file
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

or this (if your awk supports multi-char RS, e.g. GNU awk):

$ awk -v RS='^$' '{gsub(/-\n/,""); $1=$1} 1' file
Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accusam et justo duo dolores et ea rebum. - Stet clita kasd gubergren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

may be what you want.

-1
awk '{gsub(/-$/,"",$0)}ORS=" "{print $0}' filename

output

Lorem ipsum dolor sit amet, consetetur sadipscing elitr, sed diam nonumy eirmod tempor invidunt ut labore et dolore magna aliquyam erat, sed diam voluptua. - At vero eos et accu sam et justo duo dolores et ea rebum. - Stet clita kasd guber gren, no sea takimata sanctus est Lorem ipsum dolor sit amet.

command: perl -pne 's/-$//g;s/\n//g' filename

output

Lorem ipsum dolor sit amet,consetetur sadipscing elitr,sed diam nonumy eirmod temporinvidunt ut labore et doloremagna aliquyam erat, sed diamvoluptua. - At vero eos et accusam et justo duo dolores et earebum. - Stet clita kasd gubergren, no sea takimata sanctusest Lorem ipsum dolor sit amet
1
  • 3
    Your awk solution puts a space in the words "accusam" and "gubergren". Your perl solution leaves no space between a word at the end of the line and the next word at the start of the next line, e.g. "amet,consetetur" and "diamvoluptua." – Kusalananda May 2 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.