2

I've been trying to figure out how to delete millions of files from a mounted NAS drive. The OS I'm accessing it from is RHEL 7.6.

The directory is actively being written to, with tens or hundreds of thousands of additional files per day.

When I try running a deletion technique, such as:

rsync -av --delete empty_dir/ millions_of_files_dir/

It will use lots of CPU/MEM at first, with the "sending incremental file list" message. This continues forever, with CPU/MEM usage slowly petering down to a minimal amount.

My theory is that this is happening because the file list never ends; there are endlessly more files being added.

SO: Is there a deletion technique which doesn't require a full list of files before deletion, but will "stream-delete" as it finds them?

  • 1
    That sounds like a pre-3.x version of rsync (10+ years old). – jordanm Feb 20 at 22:24
  • Just checked, it's 3.1.2. – Logic Crypto Feb 20 at 22:54
  • Does this answer your question? Command to Delete Millions of Files From Directory – roaima Feb 20 at 23:03
  • 1
    2. Can you log in to the NAS and delete the files locally? – roaima Feb 20 at 23:05
  • 1
    3. What about mv millions_of_files_dir DELETE.ME && rm -rf DELETE.ME – roaima Feb 20 at 23:06
7

Using the find command with the -delete option, files will be deleted as they are found, without building a full list:

find . -type f -delete # deletes all files in the current dir and all subdirs

Something to note is that most Linux filesystems perform poorly with that many files in a single directory. Even without building a "full" list, something will still have to list the contents of the directory, which will have terrible performance if it contains millions of files.

| improve this answer | |
  • 2
    At least find should read directories in natural order. ls usually reads the whole directory first, just so it can sort the listing. ls -U avoids this. However, even if you delete in directory order, it is likely in most file systems that entries will be re-used as soon as you delete them. It seems likely this one-directory mechanism is untenable in the long run, and switching to a midnight directory rename to partition the data would be a great help. – Paul_Pedant Feb 20 at 22:58

Not the answer you're looking for? Browse other questions tagged or ask your own question.