4

I've got a directory tree created by rsnapshot, which contains multiple snapshots of the same directory structure with all identical files replaced by hardlinks.

I would like to delete all those hardlink duplicates and keep only a single copy of every file (so I can later move all files into a sorted archive without having to touch identical files twice).

Is there a tool that does that?
So far I've only found tools that find duplicates and create hardlinks to replace them…
I guess I could list all files and their inode numbers and implement the deduplicating and deleting myself, but I don't want to reinvent the wheel here.

  • 1
    find . ! -type d -links +1 finds files that are linked to more than one directory. – Stéphane Chazelas May 31 '16 at 14:51
2

To find the inodes that have more that one link:

 find . -exec stat -c '%i' {} \; | sort -n | uniq -d

Then you can iterate that list with

 find -inum {inode_number}

to list the files that share the inode. Which to remove is up to you.

2

In the end it wasn't too hard to do this manually, based on Stéphane's and xenoid's hints and some prior experience with find.
I had to adapt a few commands to work with FreeBSD's non-GNU tools — GNU find has the -printf option that could have replaced the -exec stat, but FreeBSD's find doesn't have that.

# create a list of "<inode number> <tab> <full file path>"
find rsnapshots -type f -links +1 -exec stat -f '%i%t%R' {} + > inodes.txt

# sort the list by inode number (to have consecutive blocks of duplicate files)
sort -n inodes.txt > inodes.sorted.txt

# remove the first file from each block (we want to keep one link per inode)
awk -F'\t' 'BEGIN {lastinode = 0} {inode = 0+$1; if (inode == lastinode) {print $2}; lastinode = inode}' inodes.sorted.txt > inodes.to-delete.txt

# delete duplicates (watch out for special characters in the filename, and possibly adjust the read command and double quotes accordingly)
cat inodes.to-delete.txt | while read line; do rm -f "$line"; done
  • 1
    find rsnapshots -type f -exec stat -f '%i%t%R' {} + | sort -k1,1 -u | cut -f2- will give you a sorted list of ALL filenames under the rsnapshots directory, with duplicate inodes removed. You can feed that in to archiving programs (e.g. tar). BTW, many archiving or backup programs (like tar, or rsync with the -H option) already know how to handle hardlinks (i.e. storing only the hardlink, rather than another copy of the file), so this isn't even necessary for them. – cas Jun 1 '16 at 2:07
  • NOTE: the FreeBSD version of cut doesn't (yet?) suppport NUL-separated input, so the find pipeline above is only safe for filenames that don't contain newlines in them. – cas Jun 1 '16 at 2:08
1

I think you are mistaken, since you think deleting all the "other" links to a file will save space. The only space you will save is a directory entry, and even that is questionable.

All hard links to a file are equal. There are no "duplicates". Files on Linux are really identified by which filesystem they are on, and what Inode number they are on that filesystem.

So when you create a file, you create an Inode, where the blocks actually live, and you create a link to that file in some directory. That link just points at that inode. If you do a hard link from that directory entry to another place, you just create a second directory entry someplace pointing to that same file.

If you run ls -i on a file, you will see it's Inode number. If you want to find other hard links to that same inode, simply run:

find /TOP-OF-FILESYSTEM -type f -inum INODE-NUMBER

Where TOP-OF-FILESYSTEM is the mount point for that filesystem, INODE-NUMBER is the inode number for the file in question. Note that "-type f" is not mandatory but just speeds up the search since you only will look for files.

Note that running ls -il on a file also (by default) it's inode number.

You can test all of this by going to a scratch directory and creating a file, then creating another link to it:

cd ~/tmp
date > temp1
ln tmep1 temp2
ls -l temp*
  • 2
    This is all true but it doesn't answer the question. I think you missed the point: this isn't about disk space, it's about processing the files without processing the same file twice under different names. – Gilles May 31 '16 at 21:54
  • 1
    Thanks for the explanation, but I'm already familiar with how hardlinks work and that they don't consume any noteworthy amount of disk space — like I said I just wanted to prune my old rsnapshot directory to only keep a single link to each inode, so I won't end up looking at the same file twice when I go through the data and sort it into my new archive. – n.st May 31 '16 at 22:04
  • Okay, that was not very clear. :) But most backup programs (even tar) understand hard links and don't waste space. But I'm glad you solved your problem. – Lee-Man Jun 1 '16 at 2:04
1

rmlint will find and remove duplicates, including hardlinks. At the moment it has no options to remove only hardlinks. Removal is done via an autogenerated shell script, so you can review that script prior to deletion.

In general be careful when using duplicate file detectors in hardlink mode (eg fdupes -H) since they can sometimes mistakenly identify a file as its own duplicate (see "path doubles" discussion here).

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.