0

I started out writing a simple bash shell script to find duplicate files within a given directory using SHA1 to detect matches. Everything worked until I encountered file names with spaces. The detection still works, but in the output the spaces are converted to linefeeds.

The script...

#!/bin/bash
export TARGET=$1
find $TARGET -type f -exec openssl sha1 \{\} \; > ./dupes.txt
COUNT=-1
for EVALUATION in `cat ./dupes.txt | sed 's/SHA1(\(.*\))\= \(.*\)$/\2 \1/' | awk '{print $1}' | sort | uniq -c | sort -nr`
do
    if [[ $COUNT == -1 ]]
    then
        COUNT=$EVALUATION
    else 
        HASH=$EVALUATION
        if [[ $COUNT == 1 ]]
        then
            break
        fi
        echo "--- duplicate set ---"
        for FILE in `grep $HASH ./dupes.txt | awk -F"[()]+" '{print $2}'`
        do
            echo "$FILE"
        done
        echo "---------------------"
        COUNT=-1
    fi
done

Run the script like...

./dupes.sh /home/dacracot/testDupes

It will create a file dupes.txt that looks something like...

SHA1(/home/dacracot/testDupes/lP3wj.jpg)= 324d91f412745481ed38aa184e5a56bfc3bf43b5
SHA1(/home/dacracot/testDupes/1673.gif)= 9c4029ec2e310f202b413d685209373d234e5465
SHA1(/home/dacracot/testDupes/.DS_Store)= b0ae6631a1412863f958da64091f4050005bf8d6
SHA1(/home/dacracot/testDupes/tae 2.svg)= 3ddc4fd6ae505bd01f370d0a018ef1f84b4d8011
SHA1(/home/dacracot/testDupes/tae.graffle)= 77f1ad6d695d944abacfe3a7f196be77125b6ef6
SHA1(/home/dacracot/testDupes/tae.svg)= 3ddc4fd6ae505bd01f370d0a018ef1f84b4d8011
SHA1(/home/dacracot/testDupes/22402_graph.jpg)= 24e5a25c8abf322d424dd5ce2e5b77381cd001c4
SHA1(/home/dacracot/testDupes/forwardcont.jpg)= 981e75060ae8e3aad2fe741b944d97219c8ccbe5
SHA1(/home/dacracot/testDupes/tae.svg.gz)= 922af5a5adbf7a4e7fd234aac7bcee2986133c4d
SHA1(/home/dacracot/testDupes/Alt2012.pdf)= 97d1fd997df9eb310b30a371c53883f5227cf10a
SHA1(/home/dacracot/testDupes/vcBZ8.jpg)= 7553c19fcb6aa159aada2e38066b5ba84465ee57
SHA1(/home/dacracot/testDupes/derm.graffle)= 0e1c4032f5f1fadc3a1643b2b77f816011c2d67f
SHA1(/home/dacracot/testDupes/WA.png)= 0e2e77624c3a76da4816f116665a041f6bdced2d
SHA1(/home/dacracot/testDupes/DRAW.GIF)= 6a8e4a2bf413e84140a0edeb40b475a5d3e4c255
SHA1(/home/dacracot/testDupes/crazyTalk.gif)= 1d938bbcb8cf09f30492df4504a50348cef7ea9d

And finally output that looks like...

--- duplicate set ---
/home/dacracot/testDupes/tae
2.svg
/home/dacracot/testDupes/tae.svg
---------------------

But as you can see from the first file, the output should be...

--- duplicate set ---
/home/dacracot/testDupes/tae 2.svg
/home/dacracot/testDupes/tae.svg
---------------------

What is changing the space to a linefeed?

0

Here's a simpler example that illustrates the issue:

$ cat input.txt
line one
line two
line three
$ for word in $(cat input.txt) ; do echo $word ; done
line
one
line
two
line
three

The $(cat input.txt) splits its input on whitespace. (Incidentally, in bash you can replace that with $(<input.txt)).

You can use the read built-in command instead:

$ while read line ; do echo "$line" ; done < input.txt
line one
line two
line three

(Since you're using awk anyway, you might consider rewriting the whole thing in awk, or in some other scripting language.)

  • 1
    And except for trimming whitespace (or other $IFS) and mangling backslashes, and dropping an unterminated last line, and usually dropping any line consisting of -n, while read x;do echo "$x"; done <file does the same thing as cat file -- and ... < <(cmd) does the same thing as cmd. (A read that doesn't iterate until EOF can differ from cat etc by leaving the input positioned for subsequent reading if there is any.) Although for the other loop while read count hash; do ... done would be much simpler than the for X ...; alternate handling as count or hash; done – dave_thompson_085 May 23 at 4:24
  • (Though as you say, awk -- or perl or similar -- can everything.) – dave_thompson_085 May 23 at 4:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.