0

I have a backup system based on rsync --link-dest. I'm trying to optimize disk usage a bit more.

Once in a while, I delete large files in the source folder. What'd be an easy way to automatically search for these files so I can delete them in the backup (if I deem it appropriate) ?

Something like "find files in /media/backup that are larger than 100MB and which aren't hardlinked to anything in /media/backup/latest"?

Notes:

  • There can be 10-20 backup folders in /media/backup, named by date. latest is a symlink to... you guessed it.
  • Files are sometimes moved around and/or renamed, and hardlinks are re-created in thoses cases by occasional checks with hardlink. So I'm assuming that the search must be done by inode, not by filename.
  • I know that some tools are aware of hardlinks, like du - du /media/backup/* will only report the extra spare used by each backup folder - but I don't know how to leverage this with find.
0

All right... time to program it then.

Here's something that works for my use-case. I'm fully aware that this is crappy code... but posting in case it can help someone else. Performance is better than I expected.

#!/usr/bin/python3

import sys
import os

if len(sys.argv) < 3:
    print("Usage: %s <basepath> <minsize>" % (sys.argv[0],))
size = int(sys.argv[2])

# step 1: find inodes of all large files in most recent backup
inodes = []
for root, dirs, files in os.walk(sys.argv[1]+"latest"):
    for name in files:
        file = os.path.join(root, name)
        statinfo = os.stat(file)
        if(statinfo.st_size >= size):
            inodes.append(statinfo.st_ino)

# step 2: find inodes of all large files in all backups, and display if only not in list built in step 1
for root, dirs, files in os.walk(sys.argv[1]):
    for name in files:
        file = os.path.join(root, name)
        statinfo = os.stat(file)
        if(statinfo.st_size >= size):
            try:
                position=inodes.index(statinfo.st_ino)
            except ValueError:
                print("Match: %s (%dMB)" % (file,(statinfo.st_size/(1024*1024)),))

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.