0

There must be an "easy" way to do this, but I can't figure out what it is.

Assume you have a plain text "file.txt" which has lines in this format (md5 sums followed by filenames):

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
365a6d8b18cab348d92db610dfc46264 bar.txt
ae42d992bf622bdc425d37b04ec9c2d5 mini.txt
b8e9ff5502d5dbe38b3fd5e3363caacf tyrion.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt
542ed609dfc4d0cae44c4b7be6d66382 mba.txt
310ee92ebc69ed79c1837fc53983b7f8 mini luoma.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

I would like to sort file.txt and have the output:

  1. Only show me lines if the md5 sum indicates the files are duplicates
  2. Put a blank line between each "group" of duplicates.

so it would look like this:

542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

(In the real case, it could be 2 duplicates or 10, or more.)

I'm guessing there might be a ruby or python guru out there who can figure this one out, but I'm open to pretty much any practical solution out there.

  • You do realize this is not enough to rule out hash collisions. Better to add another indicator like file size, when creating your list. Sure, you might still get collisions but the probability is much lower. – Dani_l Jul 31 '19 at 2:55
  • The better solution would be to use a deduplicating filesystem like zfs, and have it maintain single copy of files under the hood (converting duplicates to hardlinks) – Dani_l Jul 31 '19 at 2:59
  • let's assume that zfs isn't a practical solution – TJ Luoma Jul 31 '19 at 5:47
4
$ sort file.txt \
| grep -f <(cut -d' ' -f1 file.txt | sort | uniq -d) \
| awk 'last && last != $1 { printf "\n" }; { last=$1 ; print}'

542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

(Thanks to "cas" for the awk suggestion.)

(And thanks to "steeldriver" for noticing a bug.)

|improve this answer|||||
  • 1
    +1. and just pipe the output to awk 'last && last != $1 { printf "\n" }; { last=$1 ; print}'. This prints a blank line whenever variable last is not empty and is different to the current $1. feel free to add this to your answer :) – cas Jul 31 '19 at 1:41
  • @cas, thanks. I'd just come up with something similar, but not quite as nice: awk 'BEGIN { x="" }; { if (x != $1) print(""); print; x=$1 }'. It's amazing how much one forgets in only a few years, and even more amazing how quickly one relearns it. – Ray Butterworth Jul 31 '19 at 1:58
  • an alternative could be grep -f <(cat file | sort | awk '{print $1}' | uniq -d) file – guillermo chamorro Jul 31 '19 at 3:10
  • 1
    Doesn't the grouping here rely on the order in which entries happen to get grepped from the original file? it could be fixed by sorting that as well e.g. replacing file.txt by <(sort file.txt) in shells that support process substitution. – steeldriver Jul 31 '19 at 11:14
  • @steeldriver. Yes, that was indeed a bug. – Ray Butterworth Jul 31 '19 at 13:36
3

With sort and uniq

sort file.txt | uniq -w 32 --all-repeated=separate
542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt
  • -w 32 limits uniq comparison to the MD5 hash (so it considers equals lines with same MD5)
  • --all-repeated=separate tells it to show duplicates, while separating groups with a blank line
|improve this answer|||||
  • is that GNU uniq, to support the --all-repeated option? – Jeff Schaller Jul 31 '19 at 14:09
  • @JeffSchaller yes – xenoid Jul 31 '19 at 14:13
  • There wasn't a Linux tag, so it's worth pointing out the use of tools that might only be available on certain systems; thanks! – Jeff Schaller Jul 31 '19 at 14:14
  • @jeffschaller You mean, like gawk and perl in other answers? – xenoid Jul 31 '19 at 14:18
  • Not really; gawk is GNU awk, and perl's invocation doesn't differ across implementations. Since --all-repeated is only available in certain uniq implementations, it can avoid confusion by simply mentioning that implementation. If the question was tagged Linux, you could (more) safely assume that the uniq tool there was GNU's. – Jeff Schaller Jul 31 '19 at 14:30
2

With a perl Hash of Arrays:

$ perl -alne '
    push @{ $h{$F[0]} }, $_ 
    }{ 
    for $k (sort keys %h) {
      @a = @{ $h{$k} }; 
      print join "\n", @a, "" if $#a  > 0
    }
' file.txt
542ed609dfc4d0cae44c4b7be6d66382 mba.txt
542ed609dfc4d0cae44c4b7be6d66382 tyrion final.txt

5ee434a2ebcf4c3c98ee07e9c1efddc0 foo.txt
5ee434a2ebcf4c3c98ee07e9c1efddc0 imac.txt

Note that this prints a trailing blank line after the last record. The sort is optional.


Similar thing in GNU awk:

gawk '
  {a[$1][NR] = $0} 
  END {
    for(k in a){
      if(length(a[k]) > 1) {
        for(v in a[k]) print a[k][v];
        print "";
      }
    }
  }   
' file.txt
|improve this answer|||||

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.