If doing the comparision line-by-line is acceptable, then the following will tell which lines are duplicated in file text
and how many times each one appears:
sort text | uniq -c | grep -vE '^\s*1 '
As an example,
$ cat text
alpha
beta
alpha
gamma
alpha
beta
$ sort text | uniq -c | grep -vE '^\s*1 '
3 alpha
2 beta
Using the usual unix tools, this could be extended, assuming the input test format is not too complex, to paragraph-by-paragraph or sentence-by-sentence comparisons.
Finding Repeated Paragraphs
Suppose that our file text
contains:
This is a paragraph.
This is another
paragraph
This is
a paragraph.
Last sentence.
The following command identifies shows which paragraphs appear more than once:
$ awk -v RS="" '{gsub(/\n/," "); print}' text | sort | uniq -c | grep -vE '^\s*1 '
2 This is a paragraph.
This uses awk
to break the text up into paragraphs (delineated by blank lines), converts the newlines to spaces, and then passes the output, one line per paragraph, to sort and uniq for counting duplicated paragraphs.
The above was tested wtih GNU awk
. For other awk
's, the method for defining blank lines as paragraph (record) boundaries may differ.
vimdiff
not do for you here?