10

A storage system had collected 1.7 billion files from revision backups over a couple of years and was getting a bit full.

So I started to delete all files older than five years. Which I assume are around 1.7 billion (!!!) files with around 90 TByte of data - I have to estimate because even a mere find or du would take weeks, maybe months. The Backend (mdraid, ext4) itself actually isn't too important because I would like to change it anyway.

I let rm delete files for one day and only got rid of around 0,1% of all files. I estimate that deleting everything that way would take one to two years. And most likely kill some drives while doing so. Not that I worry too much, it is a Hotswap RAID.

I have been using ionice -c3 to make sure files are only deleted while the drives are not busy to avoid disk thrashing as the drive usually is under heavy load for 1-2 hours per day. On a rather funny sidenote, when I tried to run rm the first times the millions of hard links drove its memory usuage to around 100GByte then it coredumped. So I split the operation into smaller parts, works file if I only delete single subdirectories but still often spikes at 20-30GByte.

My two questions:

  1. how do I delete the old files on this system in a way that doesn't take years?

e.g. I thought about manually editing the Inode-Structures so the files are gone but the space isn't given back and then let fsck repair the system.

other mad ideas are welcome. I can always get back by making an LVM snapshot.

  1. what setups are there to avoid the same problem in the future? Eg. using a different file system, different tool chain, putting meta data (Inodes, allocation tables etcpp) on SSD - the data itself needs to stay on HD for several reason.

If noone comes up with a better idea I will just massively reduce the numbers of revisions created and/or tar/xz everything older than one month to an external USB drive. Which would be uncool because the users actually enjoy being able to access old stuff from the revisions.

9
  • 2
    +1 interesting problem: you run a BBS or some' ? What command do you use to delete files and run into the massive memory hogging you describe at this point ? My first thought was you could parallelize that job, but ... nope, this is an I/O pbm ... How about copying what you want to keep (the smaller part of what is) to a different volume (equipped w/ your favorite FS) before reformatting the raid superblocks ? Depending on file size distribution, either transfer rate or I/O will limit copy speed... Sorry just toying w/ ideas. Better to do housekeeping as you go...
    – Cbhihe
    Nov 27, 2021 at 8:52
  • 1
    Which commands have you tried (find, rm, unlink...)? Nov 27, 2021 at 9:51
  • 3
    @Bib: can you explain that ? You can be as technical as you want/need to be.
    – Cbhihe
    Nov 27, 2021 at 17:14
  • 1
    Does the filesystem have to stay online or can you unmount it? Nov 27, 2021 at 17:33
  • 1
    I wonder if ext3 debug tools might be faster. There are commandline tools which operate on an unmounted file system directly. They may be able to take shortcuts that the kernel drivers can't for for safety. Nov 28, 2021 at 14:25

2 Answers 2

4

Without access to the system and without experimentation, it would be Difficult to check what works, what helps and what is useless; but my way would be like this:

In short : Do not Delete the unwanted files, rather mv them to Directories (which must be a fast operation) and then truncate the files here to 0 size (to get the space back); later you can rm the Directories (to totally eliminate the files and get back the inodes); each of these 3 stages can be done Parallelly or Sequentially based on system load.

Details :
Make a Directory X.
In one shell script S1, mv about N=500 unwanted files into X/latest and rename that to X/X1, mv next N unwanted files into X/latest and rename that to X/X2, mv next N unwanted files into X/latest and rename that to X/X3 ....
In another shell script S2, go into each Directory X/X1 , X/X2 , X/X3 having N files and truncate the files to 0 size and rename that Directory to X/0X1, X/0X2, X/0X3 ....
In one last shell script S3, rm the Directories X/0X1 , X/0X2 X/0X3 ....

Here, the Directory Naming ensures that each shell script is independent and will not interfere with the others : S1 works with X/latest ; S2 works with X/X1, X/X2, X/X3 ... ; S3 works with X/0X1, X/0X2, X/0X3 ... : no conflicts !

Check whether each of these 3 stages can be done Parallelly or Sequentially based on system load. Vary N and use nice & ionice with sleep to control the system load.

Alternate Suggestion:
Use new location to store the newer revisions and let the users look here by Default. You could even Populate this new location (cp or mv) with revisions generated over last 1 month.
In case, one user wants "all revisions", only then access the old location.
This will ensure that the old location does not grow. Then you can easily rm unwanted very old revisions leisurely without system load.

2
  • 1
    The idea of reducing the file size is actually brilliant even though it doesn't help in my case as the files are hardlinked sometimes thousands of times - truncating one link would truncate also those I want to keep. But still I like your idea a lot. With some find-magic I have been able to only truncate files not linked in the last five years. While I guess this only works for 10-20% of the files in question it does really help. Nov 30, 2021 at 12:05
  • Oh, Nice to know that it helps atleast a little, @CrassSpektakel , & I have a Suggestion in case of retaining the other hard-links : In those cases, you could skip the truncation in X/X1 & Directly mv those files to X/0X1 where it will go through regular Deletion, leaving the other hard-links untouched !
    – Prem
    Nov 30, 2021 at 15:00
2

You can mount partition with larger commit interval (this is relatively save but may not be helpful) or with nobarrier (should help) which is extremely dangerous in terms of power loss or kernel crash.

Asynchronous I/O magic may help but i can't suggest any tools.

1
  • 1
    nobarrier also helped a bit. I already had noatime - which did also speed up things considerably - so nobarrier was a nice bone. I can't see any way to use Asynchronous I/O though. Your suggestion gave me the idea toI disable the journal for a short test by using tune4fs -O ^has_journal and reactivated it later. This also helped considerably. I am just not sure I really want to try that for a week long task, same goes for nobarrier. Time will tell. Nov 30, 2021 at 12:13

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .