7

I have an assignment to put sentences in a text file on separate lines. Something like this almost works:

cat file.txt | tr '.' '\n'

But I don't want to lose dots, question marks and exclamation marks from my sentences. How can I make this work?

  • 2
    Please show us an example of your input file and your desired output. – terdon Mar 18 '15 at 14:19
  • 2
    Are you worried about line breaks after punctuation marks in sentences? E.g., Latin abbreviations (e.g., 'e.g.'), quoted sentences (e.g. '"What!", they exclaimed), notations (e.g., '3! is six.') – Charles Stewart Mar 18 '15 at 19:37
  • 3
    This is a very hard thing to get right. You need either to define the scope of the question much more narrowly and precisely, or develop a program that can really understand English. Three challenges that none of the answers so far have met: (1) The abbreviation “Dr.” might be tied to the next token (word/name) or the previous one. Given the text, “Dr. Oz lives on Lakeside Dr. Phil is homeless.”, it’s very hard to realize that “Phil” begins a new sentence. … (Cont’d) – Scott Mar 18 '15 at 22:39
  • 1
    (Cont’d) … (2) Quotes and parentheses. He asked, “What should I bring?” She replied, “A bottle of wine would be nice.” Then she hung up. Correct behavior is to break after the after the . or ?. Likewise, “Use awk. (It’s POSIX-compliant.) Or use mygawk.”, where you need not to break between the “compliant.” and the “)”. (3) Sometimes “...” occurs within a sentence. Also, (4) Given the text, “Oh! I forgot to turn off the stove.”, some people might consider this to be one sentence; “Oh!” is clearly not a sentence, as it contains neither subject nor verb. But that’s subjective. – Scott Mar 18 '15 at 22:41
  • I see, ex post facto, that Charles already addressed some of these points. +1 for “n! represents n factorial.” – Scott Mar 18 '15 at 22:42
17

I can't be sure without seeing an actual example of your data but what you're probably looking for is adding a newline after each occurrence of .,! and ?. I don't know how you want to deal with semicolons (;) since they're not really marking an end of a sentence. That's up to you.

Anyway, you could try sed:

$ echo 'This is a sentence! And so is this. And this one?' | 
    sed 's/[.!?]  */&\n/g' 
This is a sentence! 
And so is this. 
And this one?

The s/// is the substitution operator. Its general format is s/pat/replacement and it will replace pat with replacement. The g at the end makes it run the replacement on all occurrences of pat. Without it, it would stop at the first one. The & is a special sed construct which means "whatever was matched". So, here we're substituting any of .,!, or ? with whatever was matched and a newline.

If your text can include abbreviations such as e.g., you might want to only replace if the next letter is a CAPITAL:

$ echo 'This is a sentence! And so is this. And this one? Negative, i.e. no.' | sed 's/\([.!?]\) \([[:upper:]]\)/\1\n\2/g' 
This is a sentence!
And so is this.
And this one?
Negative, i.e. no.

Note that this will not deal with sentences like Dr. Jones said hello. correctly since it will assume that the . after Dr defines a sentence given that the next letter is capitalized. However, we are now approaching a level of complexity that is way beyond the simple Q&A format and actually requires a full-blown natural language parser.

  • 1
    +1 for being the first answerer to (a) capture (and eliminate) the space after the punctuation mark, and (b) include the g at the end (since a short sentence can be contained entirely within one line, with other stuff before and after). If I could give you +2, I would have, if only you had said (space)* -- since some of us old folks still type the way we were taught: separating sentences with two spaces. (Also, if the input text is the output of something like RUNOFF, roff, or nroff, extra space characters could be used to achieve text justification.) – Scott Mar 18 '15 at 18:56
  • Never heard of that two spaces concept, interesting – TheBlastOne Mar 18 '15 at 18:59
  • @Scott thanks, and very good point for matching space*, answer edited. – terdon Mar 18 '15 at 19:02
  • @don_crissti true, thanks. I added two spaces with the second being optional. That should work as long as a sentence is properly written (with a space following the punctuation mark) and won't add an empty line unless there's a trailing space in the input. – terdon Mar 19 '15 at 13:44
  • 2
    @don_crissti damn, I hadn't even considered abbreviations. You could do something like sed 's/\([.!?]\) \([[:upper:]]\)/\1\n\2/g' to only match if the next character is a capital letter. – terdon Mar 19 '15 at 14:19
6

Try:

sed -e :1 -e 's/\([.?!]\)[[:blank:]]\{1,\}\([^[:blank:]]\)/\1\
\2/;t1'

On an input like:

Sentence 1. Sentence 1.2? Sentence 2!? Sentence 3.
Sentence 4... Sentence 5.

It gives:

Sentence 1.
Sentence 1.2?
Sentence 2!?
Sentence 3.
Sentence 4...
Sentence 5.

(and is POSIX).

  • @mikeserv, no I repeat while it matches. Like adding the g flag to s but with the addition that it handles both .s in . . x. – Stéphane Chazelas Mar 18 '15 at 15:56
  • 1
    +1 for [[:blank:]]\{1,\}, although it would be nice if you explained what you were doing. I understand it, but the OP admits to being a newbie. – Scott Mar 18 '15 at 19:03
2

Use sed instead :

sed 's/\./\.\n/' file.txt
  • 1
    Sentences don't only end on full stops. Also, UUoC. – terdon Mar 18 '15 at 14:25
2

The task has some pitfalls. One option could be:

sed 's/\([.?!;]\) */\1\n/g' file.txt

This is substituting the characters in the given character set ([.?!;], add a colon or remove the semicolon as fits your needs) followed by optional blanks ( *) by the replaced character (\1 expands to the match between \( and \)) and a newline (\n).

  • Stephane, I think you'd have to add the \n hint to most of the solutions posted here. (I edited the \? for *.) – Janis Mar 18 '15 at 14:38
  • +1 for being the first answerer to use (space)*. – Scott Mar 18 '15 at 18:57
1

Try:

awk -F. '{ for (i=1;i<=NF;i++) printf "%s.\n",$i ;} ' < input_file > output_file

where

  • awk uses . (dot) as the separator,
  • and loops for every field, printing the line, a dot a newline
  • That only matches periods. What about ?!? Also, you need to take into account the space between the period and the next sentence. – terdon Mar 18 '15 at 14:26
  • OP mention only dot. – Archemar Mar 18 '15 at 14:32
  • No they didn't: ", I don't want to lose dots, question marks and exclamation marks from my sentences." – terdon Mar 18 '15 at 15:25
  • What do you mean by "a dot a newline"? "one dot per newline"? – Peter Mortensen Mar 18 '15 at 17:19
  • 1
    @Peter: In case you haven't figured this out yet, I suspect that Archemar meant "printing the line, a dot , and a newline." – Scott Mar 20 '15 at 23:44
1

There is life outside one-liners...

Sentence spliters are never ready, there is always one more detail to fix: a Perl multiliner!

#!/usr/bin/perl

use strict;
my $pont=qr{[.!?]+};                   ## pontuation
my $abrev=qr{\b(?:Pr|Dr|Mr|[A-Z])\.};  ## abreviations

$/="";   

while(<>){ chomp;                      ## for each paragraph,

  s/\h*\n\h*/ /g;                      ## remove \n
  s/($pont)\h+(\S)/$1\n$2/g;           ## pontuation+space
  s/($abrev)\n/$1 /g;                  ## undo \n after abreviations

  print "$_\n\n";
}

so with:

A single ‘-’ operand is not really an option ! It stands for
standard input. Or for standard output ? For example:
‘smth -’ reads from stdin; and is equal
to plain ‘smth’... Could it appear as any operand that
requires a file name ? Certainly !

Robert L. Stevenson wrote  Dr. Jekyll and Mr. Hyde. Back in 12.12.1886

the end

the output is:

A single ‘-’ operand is not really an option !
It stands for standard input.
Or for standard output ?
For example: ‘smth -’ reads from stdin; and is equal to plain ‘smth’...
Could it appear as any operand that requires a file name ?
Certainly !

Robert L. Stevenson wrote  Dr. Jekyll and Mr. Hyde.
Back in 12.12.1886

the end
  • Very good! You’re the only person who has attacked the thorny issue of abbreviations (e.g., “Mr. Spock”). But (1) It’s nearly impossible to create a complete list of abbreviations that end with a period. Obvious ones you missed include “Mrs”, “Ms” (a woman of unspecified marital status), “St” (saint), and “Prof” (professor). I don’t recognize “Pr” — do you use that for “professor”? … (Cont’d) – Scott Mar 18 '15 at 22:58
  • (Cont’d) … A few others you missed are military ranks (Gen, Col, Maj, Capt, …), “Drs” (the plural of “Dr”, as in “Drs. Oz and Phil”), compass directions (N, S, E, W), and “No” meaning either “North” or “number”. (2) Conversely, your code assumes that “Dr.” is never the end of a sentence, so “Oak St. becomes Lakeside Dr. The Lake Chalet is on your right.” is rendered as “Oak St.” // “becomes Lakeside Dr. The Lake Chalet is on your right.” … (Cont’d) – Scott Mar 18 '15 at 22:59
  • (Cont’d) … (3) Likewise, it assumes that a single letter can never be the last word of a sentence. Consider, “Who is going? John and I. When? Tomorrow.” Your code thinks that “John and I. When?” is all one sentence. Likewise, “The 17th letter of the alphabet is Q. The 18th letter of the alphabet is R.”, “Shakespeare wrote a play about Henry V. …” “Have you heard of Malcolm X? …” (4) See also my comment on the question. – Scott Mar 18 '15 at 23:00
  • @Scott, thank you for the comments, bug reports. As I claimed in the beginning: they are never ready. The (long) way to go is try to increase the precision step by step, knowing that we are far from 100%. My sentSplitter for portuguese corpora has many more lines (including options to process LaTex math, Chess notation games, urls, emails, etc). This task clearly needs testing, evaluation, learning techniques..., multi function, modularity, etc – JJoao Mar 18 '15 at 23:31
-1
sed 's/\([.!?]  *\)\{0,1\}/\1\\/g' <infile | xargs printf %s\\n

I had this thing w/ fold before - which was fast - but i realized i could do the same w/ xargs w/out having to preprocess input or to implement any sed branch loops if i just backslash-escaped everything that wasn't a terminating char or any subsequent spaces.

So in the above statement sed will match either the null string or a terminating sequence for every character (not in the sequence) that occurs in input. In the right-hand side sed substitutes either the null-string or the terminating sequence in for \1 and afterward inserts a backslash. The result is that every char but one of .!? when followed by at least one space gets \backslash-escaped. This includes the \newline that sed inserts after each substitution when it writes to stdout.

Because xargs will elide entirely a backslash-escaped \newline and will split the args it hands its named utility on unescaped spaces, printf winds up printing all sentence-like strings which xargs reads in input sans any trailing spaces and on a single line per. What's more - it does it in batches approaching ARGMAX size - as many as possible at a time. And of course sed should manage its job pretty quickly as well - it only has to do the one global substitution per input line.

The results work out like this:

Some Lorem Ipsum taken from www.lipsum.com:

sed 's/\([.!?]  *\)\{0,1\}/\1\\/g' <<LIPSUM | xargs printf %s\\n
Section 1.10.32 of "de Finibus Bonorum et Mal
orum", written by Cicero in 45 BC
"Sed ut perspiciatis unde omnis iste natus er
ror sit voluptatem accusantium doloremque lau
dantium, totam rem aperiam, eaque ipsa quae a
b illo inventore veritatis et quasi architect
o beatae vitae dicta sunt explicabo.
Nemo enim ipsam voluptatem quia voluptas sit 
aspernatur aut odit aut fugit, sed quia conse
quuntur magni dolores eos qui ratione volupta
tem sequi nesciunt.
Neque porro quisquam est, qui dolorem ipsum q
uia dolor sit amet, consectetur, adipisci vel
it, sed quia non numquam eius modi tempora in
cidunt ut labore et dolore magnam aliquam qua
erat voluptatem.
Ut enim ad minima veniam, quis     ...
...
reiciendis voluptatibus maiores alias consequ
atur aut perferendis doloribus asperiores rep
ellat."        1914 translation by H.

LIPSUM

...which prints...

Section 1.10.32 of "de Finibus Bonorum et Malorum", written by Cicero in 45 BC
"Sed ut perspiciatis unde omnis iste natus error sit voluptatem accusantium doloremque laudantium, totam rem aperiam, eaque ipsa quae ab illo inventore veritatis et quasi architecto beatae vitae dicta sunt explicabo.
Nemo enim ipsam voluptatem quia voluptas sit aspernatur aut odit aut fugit, sed quia consequuntur magni dolores eos qui ratione voluptatem sequi nesciunt.
Neque porro quisquam est, qui dolorem ipsum quia dolor sit amet, consectetur, adipisci velit, sed quia non numquam eius modi tempora incidunt ut labore et dolore magnam aliquam quaerat voluptatem.
Ut enim ad minima veniam, quis     ...
...
reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat."        1914 translation by H.
  • @don_crissti - yeah - weird. It did work w/ busybox xargs. And it does w/ read, but not otherwise. It takes very little more to more surely quote it - I guess I have to. Or else put a tr -d \\n between the sed and xargs. (I just did it in the shell on my android tablet in the bathroom, honestly) – mikeserv Mar 19 '15 at 21:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.