-4

No fdup please..I want to make a script.

I have a lot of file duplicates,there are more than 200

I made ( is under construction ) a bash script which make

md5sum on every file,then with uniq i put the duplicate md5s on

another file,then again i check the duplicates and put the entire line

in a third final file.

Now the problem..i can remove one by one the dups.

But my question is: is possible to find only the dups and put on 4th file to delete safely?

This is the script

#!/bin/bash

# Script is "under construction"

# First we make the md5sum
find mp3 -type f -print0 |xargs -0 md5sum|tee firstfile.txt

# Then we find all the md5sum identical and put in secondfile.txt
sort +1rn -2 +0 -1 firstfile.txt |awk '{print $1}'|uniq -d > secondfile.txt


# then we extract from the secondfile and firstfile md5sum and name
while read line;do grep -i $line firstfile.txt;done  < secondfile.txt > thirdfinal.txt

Now the problem..thirdfinal.txt contains a lot of lines similar to those

625e8fd5f878b19b39826db539e01cda  mp3/16.mp3
625e8fd5f878b19b39826db539e01cda  mp3/12.mp3
625e8fd5f878b19b39826db539e01cda  mp3/20.mp3
625e8fd5f878b19b39826db539e01cda  mp3/21.mp3
625e8fd5f878b19b39826db539e01cda  mp3/19.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/9.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/5.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/7.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/10.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/8.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/3.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/2.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/1.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/11.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/6.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/4.mp3
d7fbd596e86dfdb546092f34ab8ca576  mp3/25.mp3
d7fbd596e86dfdb546092f34ab8ca576  mp3/25.mp3

My question is...how to grep the third file to obtain

a 4th file which includes ALL duplicates..except the first

line(otherwise you delete ALL file including the original!)

So you can have a line of duplicates but at the same time

preserve the original one.

The 4th file must appear like this

625e8fd5f878b19b39826db539e01cda  mp3/12.mp3
625e8fd5f878b19b39826db539e01cda  mp3/20.mp3
625e8fd5f878b19b39826db539e01cda  mp3/21.mp3
625e8fd5f878b19b39826db539e01cda  mp3/19.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/5.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/7.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/10.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/8.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/3.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/2.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/1.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/11.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/6.mp3
7eac02c26935323fe167d6e39ef6bd0a  mp3/4.mp3
d7fbd596e86dfdb546092f34ab8ca576  mp3/25.mp3

Suggestions?Please don't answer : use fdup or external programs..I

prefer bash automation

  • 1
    Please format your question for readability and include your attempts so far. – l0b0 Jan 15 '18 at 0:32
  • 1
    I noticed that you have the same file path listed twice, meaning you could end up removing your only copy of mp3/25.mp3 – Jeff Schaller Jan 15 '18 at 0:38
  • 2
    If you don't want to use external programs why are you yourself using find, awk, md5sum, etc.? What is wrong with using a tool such as fdupes? – roaima Jan 15 '18 at 1:27
  • Also, paragraphs are a good thing. Please use them instead of this half sentence per line nonsense. And how many problems do you have? grep -c 'Now the problem' – muru Jan 15 '18 at 1:35
3
awk '{ if (seen[$1]++) print }' < file3 > file4

This builds up an awk array of the md5sums in column 1; if the array value for a particular md5sum has already been seen (e.g. not the first time it is seen), then it prints the line. Either way, it increments the array value for that md5sum, starting from the default of zero.


Another way, using bash associative arrays:

unset md5sums
declare -A md5sums
while read md5sum path
do 
  ((md5sums[$md5sum]++))
  [[ ${md5sums[$md5sum]} -gt 1 ]] && printf "%s %s\n" "$md5sum" "$path" 
done < file3 > file4

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.