8

I'm currently trying to solve an issue using fdupes. I'd like to compare two folders with each other, and afterwards delete all duplicate files within one of these directories.

Example:

Files are being stored automatically in /srv/—lots of duplicates there. They shall be all left untouched. I also have a dir called /watchfolder/ and I want to remove all files in watchfolder if they are existent in /srv/.

I've tried fdupes -r srv/ watchfolder/ and the other way around. But it keeps messing with my files in srv/.

2 Answers 2

5

Filter When Recursing with Fdupes

If you have more than one duplicate then you might end up with something like:

srv/foo                               
srv/a/b/foo
watchfolder/foo
watchfolder/c/foo
watchfolder/d/foo

In such a case, you need to feed the list of duplicates into a filter or shell script to apply some smarter rules, unless you only want to preserve the very first duplicate found (e.g. the least deeply-nested match in srv). If that's all you want, then:

fdupes --recurse --delete srv/ watchfolder/

would work. For more complex situations, such as wanting to preserve everything in srv/, consider a filter like:

fdupes --recurse srv/ watchfolder/ | sed '/^srv/d; /^$/! s/.*/"&"/' | xargs rm
1
  • 2
    I wouldn't consider last example as a good practice. If watchfolder/ contains duplicates of file which copy is not contained in srv/ it leads to data loss. Be careful and make backups before that action.
    – Bombazook
    Jul 6, 2016 at 15:09
3

fdupes will keep the first file, as in the file with the earliest timestamp. It is a bit misleading in what it states in the help.

$ ll foo/ bar/
bar/:
total 12
-rw-rw-r--. 1 BriGuy BriGuy   2 Jul 23 16:10 a
-rw-rw-r--. 1 BriGuy BriGuy 102 Jul 23 16:22 b
-rw-rw-r--. 1 BriGuy BriGuy 610 Jul 23 16:23 c

foo/:
total 12
-rw-rw-r--. 1 BriGuy BriGuy   2 Jul 23 16:10 a
-rw-rw-r--. 1 BriGuy BriGuy 102 Jul 23 16:11 b
-rw-rw-r--. 1 BriGuy BriGuy 610 Jul 23 16:22 c

$ fdupes foo/ bar/
foo/b                                   
bar/b

foo/c
bar/c
# in above foo/b and foo/c would be kept

$ cp bar/c foo/c
$ fdupes foo/ bar/
bar/c                                   
foo/c

foo/b
bar/b
# in above foo/b and bar/c would be kept,
# as bar/c has an earlier timestamp than foo/c now
1
  • 2
    A good addendum to this (good) answer would also be the case when bar and foo get swapped: $fdupes bar/ foo/. Because, unlike you might think, the output is likely to be the very same as in $fdupes foo/ bar/, since timestamp is the only thing it cares for, as you correctly pointed out. This can drive you totally nuts if you want to keep one folder as-is no matter what. And since fdupes can only protect the first file (top-down), the order WILL be important. Anyways, I consider any method involving grep or sed an ugly workaround for a badly-conceived tool. Oct 28, 2015 at 19:06

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .