chmod recursive permission on thousands of files

Question

This is a more general question about 'chmoding' recursively.

I have this script which at some point needs to change the permissions recursively in a folder which has a few hundred thousand files. There are new files added in that folder every day, but the ones that are already there have the permissions already set and they don't change.

My question is... when I call

chmod 775 . -R 

does it try to set the permission for the files that already have the right permissions set, or only for the new files that don't have the right permissions?

It seems to always take ages to get past this command in the script, even though the 'new' files are only a few thousand and it should do their permissions fairly quickly.

I've looked at the man page for chmod, but it doesn't seem to mention anything on this case.

If chmod doesn't check for permissions beforehand, should I start looking at combining 'find' with 'chmod'?

Answer 1

chmod might or might not change the permissions of files that are already set to what you want, but if not, it would still need to check them to see what their current permissions are[0]. With hundreds of thousands of files, I don't think it would matter either way; the time is most likely being spent by the tools stating every file.

You can try using find to either check for files newer than the last run or files that need chmod to be run, but I don't think you'll get much speed improvement.

If possible for your script, you might be able to get the new files put into a separate directory first, as a "holding" area. Then you can chmod THAT directory (which only has new files), and mv them in with the rest. That should be substantially faster, but unfortunately won't work for every application.

[0] Even if it does try to set the permission of files that don't need any changes, the underlying filesystem probably won't do anything with the request, because it's unnecessary.

Answer 2

find / chmod optimization

Both find and chmod have to read

1. all directory entries
2. the inodes for all these entries

You probably get a performance improvement by first reading all the entries and then all the inodes (on a rotating disk) because then the disk head does not move between the directory and the inodes). As chmod is stupid (as one of the other answers explains) it should be called through find only. But even then it may help to read all the inodes before the first gets written (assuming you have enough free RAM for the disk cache). I suggest this:

find . -printf "" # reading the file names only
find . ! -perm 775 -printf "" # reading all the inodes (file names are cached)
find . ! -perm 775 -exec chmod 775 + # writing to the cache without reading from disk

The good solution: ACLs

The good solution may be completely different: If the files are created in this directory (and not moved from somewhere else) then ACLs can do the job on the fly. You just have to set the default ACLs on the parent directory.

Further improvement may be reached by filesystem optimizations. If it is ext3/ext4 then you may run e2fsck -D from time to time. Maybe it helps to put this directory onto a separate volume. You may try different filesystems or filesystem settings (e.g. different inode sizes).

Answer 3

Assuming the use of chmod from the GNU coreutils package on Ubuntu 12.10.

chmod 775 . -R executes the fchmodat system call for each file that it finds irrespective of whether the permissions need changing or not. I confirmed this by both inspecting the code and using strace chmod 775 . -R (snippet below) to list the actual behaviour.

newfstatat(4, "d", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "d", 0775)                  = 0
newfstatat(4, "c", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "c", 0775)                  = 0
newfstatat(4, "a", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "a", 0775)                  = 0
newfstatat(4, "b", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "b", 0775)                  = 0

There are a couple of disadvantages of running fchmodat on each file

• The extra system call will likely become significant if a large number of files are changed. The find/xargs/chmod method mentioned by others will likely be quicker by only changing files that need changing.
• The call to fchmodat changes the file status modification (ctime) of each file. This will cause every file/inode to change each time and will likely cause excess disk writes. It might be possible to use mount options to stop these excess writes.

A simple experiment shows the ctime changes happening for straight chmod

auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 d
auser@duncow:/tmp/blah.test$ chmod 775 . -R
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

But this does not change for find/xargs/chmod a few minutes later

auser@duncow:/tmp/blah.test$ date
Tue Jun 18 18:27:27 BST 2013
auser@duncow:/tmp/blah.test$ find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {}
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

I would always tend to use the find/xargs/chmod version because find gives more control over selecting things.

Answer 4

coreutils/src/chmod.c#process_file shows that chmod(1) always tries to set the mode and then checks back again with fstatat(2).

Files are processed via fts(3), which has to stat(2) all traversed file system objects beforehand to build its data tree.

Unixlore features in Speeding Up Bulk File Operations measurements where chmod(1) is timed against an find / xargs approach: the latter wins by magnitudes.

Here the command line adapted to the original question:

find . -print0 | xargs -0 chmod 775

Two reasons:

1. File system traversal is decoupled from operations on the files via the pipe between the two processes, which might even run on different cores.

   2. fts(3) operation is minimized, because xargs(1) 'flattens' out the directory trees.

So yes: you should definitely use find / xargs for a simple solution.

Other options:

• Play with the umask and the source code of the process(es) writing the new files.

• If you are using Linux, chances are your system has enabled the inotify kernel subsystem. In this case, you can create a script with an efficient solution via inotifywait(1).

---

Sidenote: unless you want to execute permissions on your files, I'd suggest modifying the invocation as so:

find . -type f -print0 | xargs -0 chmod 664
find . -type d -print0 | xargs -0 chmod 775

---

Note to the editors: I am not allowed to add more than two links to the post, neither to comment on other posts. I leave the URLs here and hope some openhearted user with sufficient reputation puts them back into the text and deletes this paragraph.

---

Comment on priming the disk cache with find . -printf "":

This might speed up the execution of the following chmod operations, however depends on available memory and i/o load. So it might work, or not. Decoupling traversal (find) and chmod operation already provides for caching, so priming the cache might be superfluous.

Answer 5

Have you considered changing the process(es) that create the file to have them created with 0775 mode? Look at the umask value in the environment - 0002 may help.