Updated: Clarify line number requirement, some verbosity reductions
From the command line, is there a way to:
- check a file of English text
- to find repeat-word typos,
- along with line numbers where they are found,
in order to help correct them?
Currently, to help finish an article or other piece of English writing,
aspell -c text.txt is helpful for catching spelling errors. But, not helpful when the error is an unintentional consecutive repetition of a word.
There can be only one one.
$ aspell -c highlander_typo.txt
aspell is a spell-checker, not a grammar-checker, so repeat word typos are beyond its intended feature scope. Thus the result is this file passes
aspell's check because nothing is "wrong" in terms of individual word spelling.
The correct sentence is
There can be only one., the second
one is an unintended repeat-word typo.
But a different situation is for example
La la la
Here the repetition is not a typo, as these are part of an artist's song lyrics.
So the solution should not presume and "fix" anything by itself, otherwise it could overwrite intentional repeated words.
Example 3: Multi-line
He has has refused his Assent to Laws, the most wholesome and necessary for the public good. He has forbidden his Governors to pass Laws of immediate and and pressing importance, unless suspended in their operation till his Assent should be be obtained; and when so suspended, he has utterly neglected to attend to them.
Modified from The Declaration of Independence
In the above six lines,
He has has refusedshould be
He has refused, the second
hasis a repeat-word typo
should be be obtainedshould be
should be obtained, the second
beis a repeat-word typo
However, did you notice a third repeat-word typo?
... immediate and
and pressing ...
This is also a repeat-word typo because though they are on separate lines they are still part of the same English sentence, the trailing end of the line above has a word that is accidentally added at the start of the next line. Rather tricky to detect by eye due to the repetition being on opposite sides of a passage of text.
an interactive program with a process similar to
aspell -cyet able to detect repeat-words, or,
a script or combination of commands able to extract line numbers and the suspected repeat words. This info makes it easier to use an editor such as
vimto jump to the repeat words and make fixes where appropriate.
Using above multi-line
jefferson_typo.txt, the desired output would be something like:
1: has has 3: and 4: and 5: be be
1: He [has has] refused his Assent to Laws, the most wholesome and necessary 3: He has forbidden his Governors to pass Laws of immediate [and] 4: [and] pressing importance, unless suspended in their operation till his 5: Assent should [be be] obtained; and when so suspended, he has utterly
I am actually not entirely sure how to display the difficult case of inter-line or cross-line repeat-word, such as the
and repetition above, so don't worry if your solution doesn't resemble this exactly.
But I hope that, like the above, it shows:
- relevant original input's line number
- some way to draw attention to what repeated, especially helpful if the line of text is also quite long.
- if the full line is displayed to give context (credit: @Wildcard), then there needs to be a way to somehow render the repeated word or words distinctively. The example shown here marks the repetition by enclosing them within ASCII characters
]. Alternatively, perhaps mimic
grep --colors=alwaysto colorize the line's matches for display in a color terminal
- text, should stay as plain text files
- no GUI solutions please, just textual.
ssh -XX11 forwarding not reliably available and need to edit over
To try to find duplicates,
uniq came to mind, so the plan was to first determine how to get repeat-word recognition to work on a single line at first.
In order to use
uniq we would need to first convert words on a line, to becoming one word per line.
$ tr ' ' '\n' < highlander_typo.txt There can be only one one.
$ tr ' ' '\n' < highlander_typo.txt | uniq -D
This is because for
-D option, which normally reveals duplicates, input has to be exactly a duplicate line. Unfortunately the period
. at the end of the repeated word
one negates this. It just looks like a different line. Not sure how I would work around arbitrary punctuation marks such as this period, and somehow add it back after
This was unsuccessful. But if it were successful, next there would need to be a way to include this line's line number, since the input file could have hundreds of lines and it would help to indicate which line of the input file, that the repeat-word was detected on.
This single-line code processing would perhaps be part of a parent loop in order to do some kind of line-by-line multi-line processing and thus be able to process all lines in a file, but unfortunately getting past even single-line repeat-word recognition has been problematic.