Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
6 added 801 characters in body
source | link

That should be be aa lot faster;


 

In case we are talking about millions of files - you wrote it's 600 million files in total - there is something more to take into account:

Most programs, including find, read directories using the library call readdir (3). That usually uses a buffer of 32 KB to read directories; That becomes a problem when the directories, containing huge lists of possibly long filenames, are big.

The way to work around it is to directly use the system call for reading directory entries, getdents (2), and handle the buffering in a more suitable way.

For details, see You can list a directory containing 8 million files! But not with ls..


That should be a lot faster;


 

That should be a lot faster;


In case we are talking about millions of files - you wrote it's 600 million files in total - there is something more to take into account:

Most programs, including find, read directories using the library call readdir (3). That usually uses a buffer of 32 KB to read directories; That becomes a problem when the directories, containing huge lists of possibly long filenames, are big.

The way to work around it is to directly use the system call for reading directory entries, getdents (2), and handle the buffering in a more suitable way.

For details, see You can list a directory containing 8 million files! But not with ls..


5 Add tree traversal reference
source | link

That is closely related to the symtoms you see; The option -depth is changing the tree traversaltree traversal algorithm for the file system tree from an preorder depth-first search to an inorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first. For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

That is closely related to the symtoms you see; The option -depth is changing the tree traversal algorithm for the file system tree from an preorder depth-first search to an inorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first. For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

That is closely related to the symtoms you see; The option -depth is changing the tree traversal algorithm for the file system tree from an preorder depth-first search to an inorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first. For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

4 added 368 characters in body
source | link

That is actually a really interesting issue... or more like..., honestly, mallicious:

Not sure how you tested, but theThe command

is very different from the test version withoutusual tryout variant, leaving out the dangerous part, -delete:

That is closely related to the symtoms you see; As depthThe option -depth is changing the tree traversal algorithm for the file system tree from an from breadthpreorder depth-first search to an depthinorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first.   For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

Find has to keep track of some directories to visit later, which is not a problem for a few directories.
But maybe with many directories, for various degrees of many.
Also, performance problems outside of find will get noticable in this kind of situation; So it is possible it's not even findfind that's slow, but something else entirely, or partially.

The performance and memory impact of that depends on your directory structure etc.
(See below for a note asking to add some data) 

That is actually a really interesting issue... or more like... mallicious:

Not sure how you tested, but the command

is very different from the test version without -delete:

That is closely related to the symtoms you see; As depth changing the tree traversal algorithm for the file system tree from breadth-first search to depth-first search, find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first.  

Find has to keep track of some directories to visit later, which is not a problem for a few directories.
But maybe with many directories, for various degrees of many.
Also, performance problems outside of find will get noticable in this kind of situation; So it is possible it's not even find that's slow, but something else entirely, or partially.

The performance and memory impact of that depends on your directory structure etc.
(See below for a note asking to add some data)

That is a really interesting issue... or, honestly, mallicious:

The command

is very different from the usual tryout variant, leaving out the dangerous part, -delete:

That is closely related to the symtoms you see; The option -depth is changing the tree traversal algorithm for the file system tree from an preorder depth-first search to an inorder depth-first search.
Before, each file or directory that was reached was immediately used, and forgotten about. Find was using the tree itself to find it's way. find will now need to collect all directories that could contain files or directories still to be found, before deleting the files in the deepest directoies first. For this, it needs to do the work of planing and remembering traversal steps itself, and - that's the point - in a different order than the filesystem tree naturally supports. So, indeed, it needs to collect data over many files before the first step of output work.

Find has to keep track of some directories to visit later, which is not a problem for a few directories.
But maybe with many directories, for various degrees of many.
Also, performance problems outside of find will get noticable in this kind of situation; So it is possible it's not even find that's slow, but something else.

The performance and memory impact of that depends on your directory structure etc. 

3 added 257 characters in body
source | link
2 added 257 characters in body
source | link
1
source | link