Tag Info

New answers tagged

1

Have you looked into PDF Content Comparer? There are command line options which should let you automate the process. You could run some sort of logic on the difference log it creates to see how similar they are. Failing that you might try splitting the PDF's into multiple files temporarily and comparing them that way. You'd probably still have duplicates ...


-1

There is an Linux application, called recoll. It can perform the task, but only for pdfs with text layer.


0

Assuming those are all *.jpg files, do a: To find the Files present: grep -Ff <(for i in </path/to/directory/a>/*.jpg ; do md5sum $i | awk {'print $1'}; done) <(find </path/to/directoryb/ -iname "*.jpg" | xargs md5sum) The for loop here creates a list of md5 checksums of all the *.jpg files in directory 'a' and the find here will create a ...


0

See the answer for a similar question from two weeks ago. find . -type f -exec md5sum {} + | sort | sed 's/ */!/1' | awk -F\| 'BEGIN{first=1}{if($1==lastid){if(first){first=0;print lastid, lastfile}print$1, $2} else first=1; lastid=$1;lastfile=$2}'


1

Since you want to compare them by content, using hashes seems to be a way to do it. You can use the find command to get a list of file paths of a directory. The -type f option will leave out all directories and only output paths to regular files. The -exec md5sum {} \; option will take the found paths and gives them to the md5sum command to turn into a list ...


1

Another simple, alternative(for someone who dont want/know awk) script will be: #!/bin/bash sort -t',' -k 3 marathon | cut -d',' -f 3 | uniq -d if someone wants to print whole line instead of just names: #!/bin/bash sort -t',' -k 3 marathon | cut -d',' -f 3 | uniq -d | grep -f - marathon in over scripts: sort takes third field to sort, using , as ...


3

You have to sort using names first. Note: 'uniq' does not detect repeated lines unless they are adjacent. You may want to sort the input first, or use `sort -u' without `uniq'. You can use the -t/-k options, to sort these fields: sort -t',' -k 3 marathon that sort regarding the 3rd field with the comma as separator. Then you can print ...


1

It is a slightly long, but it is a single command-line. This looks at the contents of the files and compares them using a cryptographic hash (md5sum). find . -type f -exec md5sum {} + | sort | sed 's/ */!/1' | awk -F\| 'BEGIN{first=1}{if($1==lastid){if(first){first=0;print lastid, lastfile}print$1, $2} else first=1; lastid=$1;lastfile=$2}' As I said, ...


0

Create directory structure mkdir dir{A,B,C} touch dirA/file{,-001,2,3}.jpg touch dirB/file{A,A_ios,B,C}.jpg touch dirC/file{X,X_ios,X-001,Y,Z}.jpg Show multiplicity of duplicate files find . -name '*.jpg' -type f |sed 's/\(.*\/\(file.\).*\(.jpg\)\)/\2/' |sort |uniq -c|grep -v 1 Returns 2 fileA 3 fileX


0

You may want to consider programs specifically intended to the search of duplicate files rather than relying on the name, e.g. fdupes or fslint.



Top 50 recent answers are included