A storage system had collected 1.7 billion files from revision backups over a couple of years and was getting a bit full.
So I started to delete all files older than five years. Which I assume are around 1.7 billion (!!!) files with around 90 TByte of data - I have to estimate because even a mere find or du would take weeks, maybe months. The Backend (mdraid, ext4) itself actually isn't too important because I would like to change it anyway.
I let rm delete files for one day and only got rid of around 0,1% of all files. I estimate that deleting everything that way would take one to two years. And most likely kill some drives while doing so. Not that I worry too much, it is a Hotswap RAID.
I have been using ionice -c3 to make sure files are only deleted while the drives are not busy to avoid disk thrashing as the drive usually is under heavy load for 1-2 hours per day. On a rather funny sidenote, when I tried to run rm the first times the millions of hard links drove its memory usuage to around 100GByte then it coredumped. So I split the operation into smaller parts, works file if I only delete single subdirectories but still often spikes at 20-30GByte.
My two questions:
- how do I delete the old files on this system in a way that doesn't take years?
e.g. I thought about manually editing the Inode-Structures so the files are gone but the space isn't given back and then let fsck repair the system.
other mad ideas are welcome. I can always get back by making an LVM snapshot.
- what setups are there to avoid the same problem in the future? Eg. using a different file system, different tool chain, putting meta data (Inodes, allocation tables etcpp) on SSD - the data itself needs to stay on HD for several reason.
If noone comes up with a better idea I will just massively reduce the numbers of revisions created and/or tar/xz everything older than one month to an external USB drive. Which would be uncool because the users actually enjoy being able to access old stuff from the revisions.