0

when i want to find duplicate lines between two files i use this command

comm -12 <(sort file1.txt) <(sort file2.txt)

or

sort file1.txt file2.txt | awk 'dup[$0]++ == 1'

But, how do I find duplicate lines in multiple files within folders. example:

mainfolder
  folder1
    file1-1.txt
    file1-2.txt
    etc
  folder2
    file2-1.txt
    file2-2.txt
    etc

and that the result in terminal is displayed by file (that is, the lines repeated in all files but specify which file is the one that contains it) to know the origin of the problem.

PD: I tried this command and it didn't work for me

file_expr="*.txt"; sort $file_expr | sed 's/^\s*//; s/\s*$//; /^\s*$/d' | uniq -d | while read dup_line; do grep -Hn "^\s*$dup_line\s*$" $file_expr; done| sort -t: -k3 -k1,2 | awk -F: '{ file=$1; line=$2; $1=$2=""; gsub(/(^[ \t]+)|([ \t]+$)/,"",$0); if (prev != "" && prev != $0) printf ("\n"); printf ("\033[0;33m%s (line %s)\033[0m: %s\n", file, line, $0); prev=$0; }'
0
3

You could do this (if no files have a tab caracter in their names):

grep -T -r . mainfolder | sort -k 2 | uniq -D -f 1

The recursive grep will output each line prefixed by the filename it is in. Then you sort based on all the fields but the first one. Finally uniq outputs just the duplicated lines, skipping the first field.

You can have more control on the files that go into sort by using find for example, or the --include and --exclude grep flags.

3
  • 2
    No need to specify "1000": if you do sort -k 2, then sort will use "from the 2nd field to the end of the line" as the key. – glenn jackman Jun 16 at 17:38
  • @glennjackman you are right, corrected. – Eduardo Trápani Jun 16 at 17:43
  • great work! thanks – ajcg Jun 16 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.