Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. Join them; it only takes a minute:

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

Updated: Clarify line number requirement, some verbosity reductions

From the command line, is there a way to:

  • check a file of English text
  • to find repeat-word typos,
  • along with line numbers where they are found,

in order to help correct them?

Example 1

Currently, to help finish an article or other piece of English writing, aspell -c text.txt is helpful for catching spelling errors. But, not helpful when the error is an unintentional consecutive repetition of a word.

highlander_typo.txt:

There can be only one one.

Running aspell:

$ aspell -c highlander_typo.txt

Probably since aspell is a spell-checker, not a grammar-checker, so repeat word typos are beyond its intended feature scope. Thus the result is this file passes aspell's check because nothing is "wrong" in terms of individual word spelling.

The correct sentence is There can be only one., the second one is an unintended repeat-word typo.

Example 2

But a different situation is for example kylie_minogue.txt:

La la la

Here the repetition is not a typo, as these are part of an artist's song lyrics.

So the solution should not presume and "fix" anything by itself, otherwise it could overwrite intentional repeated words.

Example 3: Multi-line

jefferson_typo.txt:

He has has refused his Assent to Laws, the most wholesome and necessary
for the public good.
He has forbidden his Governors to pass Laws of immediate and
and pressing importance, unless suspended in their operation till his
Assent should be be obtained; and when so suspended, he has utterly
neglected to attend to them.

Modified from The Declaration of Independence

In the above six lines,

  • 1: He has has refused should be He has refused, the second has is a repeat-word typo
  • 5: should be be obtained should be should be obtained, the second be is a repeat-word typo

However, did you notice a third repeat-word typo?

  • 3: ... immediate and
  • 4: and pressing ...

This is also a repeat-word typo because though they are on separate lines they are still part of the same English sentence, the trailing end of the line above has a word that is accidentally added at the start of the next line. Rather tricky to detect by eye due to the repetition being on opposite sides of a passage of text.

Intended output

  • an interactive program with a process similar to aspell -c yet able to detect repeat-words, or,

  • a script or combination of commands able to extract line numbers and the suspected repeat words. This info makes it easier to use an editor such as vim to jump to the repeat words and make fixes where appropriate.

Using above multi-line jefferson_typo.txt, the desired output would be something like:

1: has has
3: and
4: and
5: be be

or:

1: He [has has] refused his Assent to Laws, the most wholesome and necessary
3: He has forbidden his Governors to pass Laws of immediate [and]
4: [and] pressing importance, unless suspended in their operation till his
5: Assent should [be be] obtained; and when so suspended, he has utterly

I am actually not entirely sure how to display the difficult case of inter-line or cross-line repeat-word, such as the and repetition above, so don't worry if your solution doesn't resemble this exactly.

But I hope that, like the above, it shows:

  • relevant original input's line number
  • some way to draw attention to what repeated, especially helpful if the line of text is also quite long.
  • if the full line is displayed to give context (credit: @Wildcard), then there needs to be a way to somehow render the repeated word or words distinctively. The example shown here marks the repetition by enclosing them within ASCII characters [ ]. Alternatively, perhaps mimic grep --colors=always to colorize the line's matches for display in a color terminal

Other considerations

  • text, should stay as plain text files
  • no GUI solutions please, just textual. ssh -X X11 forwarding not reliably available and need to edit over ssh

Unsuccessful attempts

To try to find duplicates, uniq came to mind, so the plan was to first determine how to get repeat-word recognition to work on a single line at first.

In order to use uniq we would need to first convert words on a line, to becoming one word per line.

$ tr ' ' '\n' < highlander_typo.txt
There
can
be
only
one
one.

Unfortunately:

$ tr ' ' '\n' < highlander_typo.txt | uniq -D

Nothing.

This is because for -D option, which normally reveals duplicates, input has to be exactly a duplicate line. Unfortunately the period . at the end of the repeated word one negates this. It just looks like a different line. Not sure how I would work around arbitrary punctuation marks such as this period, and somehow add it back after tr processing.

This was unsuccessful. But if it were successful, next there would need to be a way to include this line's line number, since the input file could have hundreds of lines and it would help to indicate which line of the input file, that the repeat-word was detected on.

This single-line code processing would perhaps be part of a parent loop in order to do some kind of line-by-line multi-line processing and thus be able to process all lines in a file, but unfortunately getting past even single-line repeat-word recognition has been problematic.

share|improve this question
up vote 2 down vote accepted

Edited: added install and demo

You need to take care of at least some edge cases, like

  • repeated words at the end (and beginning) of the line.
  • search should be case insensitive, because of frequent errors like The the apple.
  • probably you want to restrict search only to word constituent to not match something like ( ( a + b) + c ) (repeated opening parentheses.
  • only full words should match to eliminate the thesis
  • When it comes to human language Unicode characters inside words should properly interpreted

All in all I recommend pcregrep solution:

pcregrep -Min --color=auto '\b([^[:space:]]+)[[:space:]]+\1\b' file

Obviously color and line number (n option) is optional, but usually nice to have.

Install

On Debian-based distributions you can install via:

$ sudo apt-get install pcregrep

Example

Run the command on jefferson_typo.txt to see:

$ pcregrep -Min --color=auto '\b([^[:space:]]+)[[:space:]]+\1\b' jefferson_typo.txt
1:He has has refused his Assent to Laws, the most wholesome and necessary
3:He has forbidden his Governors to pass Laws of immediate and
and pressing importance, unless suspended in their operation till his
5:Assent should be be obtained; and when so suspended, he has utterly

The above is just a text capture, but on a color-supported terminal, matches are colorized:

  • has has
  • and
  • and
  • be be
share|improve this answer
    
And you can add -o option to see only matched parts, so exactly the output you want (after question edit). – jimmij Jan 20 at 0:36
    
Great edge cases, almost as though you yourself faced this same problem too, so I'm glad I asked as this problem had been nagging me for some time. Note I updated question by adding example 3 jefferson_typo.txt about the same time you wrote this, it has the cross-lines repeat word edge case you mentioned, so I'll try installing pcregrep first and then test your solution on jefferson_typo.txt – user454038 Jan 20 at 3:09
    
I've edited your answer with install info and a demo of successful output. Even though it doesn't add a line number for the second and, that's fine since there is one for the first, works the best. If you'll accept the changes, and as long as no other better answer comes up before you accept, then I'll accept it as the best answer, thanks – user454038 Feb 9 at 2:15

You should take a peek at the venerable diction(1) and style(1) commands. They catch a variety of boo-boos. There are newish versions (GPLv3 here on Fedora 23).

Install

For example on Debian-based distributions, install the package diction, which includes style:

$ sudo apt-get install diction

At least in Fedora it is:

$ dnf install diction

Red Hat Enterprise (and clones) probably need:

$ yum install diction

In any case, this comes from an upstream GNU package called diction, so it should be called the same almost everywhere.

Example

$ diction jefferson_typo.txt
jefferson_typo.txt:1: He has [has] refused his Assent to Laws, the [most] wholesome and necessary for the public good.

jefferson_typo.txt:3: He has forbidden his Governors to pass Laws of immediate and [and] pressing importance, unless suspended in their operation till his Assent should be [be] obtained; and when [so] suspended, he has utterly neglected to attend to them.

2 phrases in 2 sentences found.

Pros

  • catches the repeated words, amongst other things

Cons

  • introduces [] markings for items not related to repeated words. For example [so], is probably marked because it can be considered extraneous per The Elements of Style by Strunk. See man diction
  • the number shown is not always the original input's line number, but is instead the line number that the sentence starts from. So for example [be] is original input's line number 5, but here it shows 3 only because [be] is a part of the sentence beginning on line 3. So this is slightly different than what you wanted
share|improve this answer
1  
Good share, upvoted. It just doesn't always give original input's line numbers with long sentences. I also couldn't find a way to suppress non-repeat-word findings from getting marked [] too. Despite this I edited your answer to share what it looks like with the jefferson_typo.txt sample input in case it can still help someone. – user454038 Feb 9 at 2:55

This will print lines (with filename and line number) with repeated words:

for f in *.txt; do
    perl -ne 'print "$ARGV: $.: $_" if /\b(\w+)\W+\1/' "$f"
done

For multi-line matching there's this, but you lose the line numbers because it's slurping in the file by paragraphs (that's the effect of the -00 option). The \W+ between the two words means any "non-word" characters, including newlines.

perl -00 -nE '
    @matches = /\b((\w+)\W+\2)/g; 
    while (@matches) {
        ($match,$word) = splice @matches, 0, 2;
        say "dup: $match";
    }
' jefferson_typo.txt 
dup: has has
dup: and
and
dup: be be
share|improve this answer
    
Upvoted, because it's good for my original simple highlander_typo.txt example. But I've just updated with a better example 3, jefferson_typo.txt. Would there be some way to adjust your solution to also include cases when the repeat word occurs across lines, as @jimmij mentions, meaning word2 is on next line? Secondly, instead of printing complete lines, is it possible to just show the specific detected word word repetition somehow? Input line can be quite long, so preferable to avoid whole line printing, but instead, just the repeated words or some other form of focused output. – user454038 Jan 20 at 2:54
1  
@user454038, given how important context is, I would think it would be useless output to just print the repeated words. The context of the line is necessary to determine whether the extra word should be removed or not. (And you should use relatively short lines; that's what fmt is for.) – Wildcard Jan 20 at 3:55

Here's what I ended up with:

tr '[:space:]' '\n' < highlander_typo.txt | tr -d '[:punct:]' |
tr '[:upper:]' '[:lower:]' | uniq -D

I believe that's GNU tr syntax.

share|improve this answer
    
Good idea about using POSIX character classes. It gets past the original stumbling block I had but I wanted to let you know I have since added a proper multi-line example, see Example 3 with jefferson_typos.txt . I think I know how I might modify your code for finding word word repetitions on a single line only, but what I also realized I need, as @jimmij has mentioned, is how to go from this, to detecting those cross-line repetitions too. – user454038 Jan 20 at 2:47
    
Why wouldn't this work for cross-line repetitions as in Example 3? All the current newlines are left unchanged. – gardenhead Jan 20 at 5:33
    
@user454038 - looks like this pipeline works for your multi-line jefferson_type.txt example too. – Bruce Ediger Jan 20 at 13:48
    
You're right about the cross-line detection, my mistake, it does detect cross-line word repetitions. So with this tr approach the only remaining task is to work in the ability to display original input's line numbers, for found instances of repeated words, to help with the editing process once repeats are identified. – user454038 Feb 8 at 22:01

Since you tagged this question with awk, why not just use awk?

$ awk '
    BEGIN{RS=FS="\\W+"}
    $0==t{printf("%s:%s\t%s %s\n", FILENAME, FNR, t, $0)}
    {t=$0}
' *.txt
highlander_typo.txt:6   one one
jefferson_typo.txt:3    has has
jefferson_typo.txt:29   and and
jefferson_typo.txt:42   be be
kylie_minogue.txt:3 la la

I didn't preserve the newline in jefferson_typo.txt because it's not visually helpful for me, but you can tweak to your tastes.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.