15

This is a more general question about 'chmoding' recursively.

I have this script which at some point needs to change the permissions recursively in a folder which has a few hundred thousand files. There are new files added in that folder every day, but the ones that are already there have the permissions already set and they don't change.

My question is... when I call

chmod 775 . -R 

does it try to set the permission for the files that already have the right permissions set, or only for the new files that don't have the right permissions?

It seems to always take ages to get past this command in the script, even though the 'new' files are only a few thousand and it should do their permissions fairly quickly.

I've looked at the man page for chmod, but it doesn't seem to mention anything on this case.

If chmod doesn't check for permissions beforehand, should I start looking at combining 'find' with 'chmod'?

  • 3
    I wonder if it's really slower to check the permissions and change them if they're not correct than directly set them to the correct value. – lgeorget Jun 18 '13 at 16:46
  • 1
    if anyone stumbles upon this and wants the find + chmod command, here it is: find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {} – Titi Dumi Jun 18 '13 at 16:46
  • @lgeorget, so you are saying it is slower to use find|chmod? than just to chmod everything. (sorry, didn't understand from your comment). cheers – Titi Dumi Jun 18 '13 at 16:48
  • In my humble opinion, it's probably slower as it needs to lauch two processes and redirect the output of the first one to the second but I'm not sure. It depends on the time it takes to set permissions which may not be that important since they are just 3 bytes to modify in the inode. – lgeorget Jun 18 '13 at 16:53
  • 1
    @depquid The main performance issue here is reading the data into the disk cache. After the first run everything is in the disk cache (unless there is too little memory) thus you are testing the performance of something which is not the bottleneck in the real situation. – Hauke Laging Jun 18 '13 at 17:25
7

chmod might or might not change the permissions of files that are already set to what you want, but if not, it would still need to check them to see what their current permissions are[0]. With hundreds of thousands of files, I don't think it would matter either way; the time is most likely being spent by the tools stating every file.

You can try using find to either check for files newer than the last run or files that need chmod to be run, but I don't think you'll get much speed improvement.

If possible for your script, you might be able to get the new files put into a separate directory first, as a "holding" area. Then you can chmod THAT directory (which only has new files), and mv them in with the rest. That should be substantially faster, but unfortunately won't work for every application.

[0] Even if it does try to set the permission of files that don't need any changes, the underlying filesystem probably won't do anything with the request, because it's unnecessary.

  • Thanks for that. I will try the find | chmod version and see if it makes things faster. If not I will try to modify the script to implement a 'holding' folder as you suggested. – Titi Dumi Jun 18 '13 at 16:56
  • The reason you wouldn't get a speed improvement is that the inode has to be read both for the ctime and the access rights. – Hauke Laging Jun 18 '13 at 16:57
11

find / chmod optimization

Both find and chmod have to read

  1. all directory entries
  2. the inodes for all these entries

You probably get a performance improvement by first reading all the entries and then all the inodes (on a rotating disk) because then the disk head does not move between the directory and the inodes). As chmod is stupid (as one of the other answers explains) it should be called through find only. But even then it may help to read all the inodes before the first gets written (assuming you have enough free RAM for the disk cache). I suggest this:

find . -printf "" # reading the file names only
find . ! -perm 775 -printf "" # reading all the inodes (file names are cached)
find . ! -perm 775 -exec chmod 775 + # writing to the cache without reading from disk

The good solution: ACLs

The good solution may be completely different: If the files are created in this directory (and not moved from somewhere else) then ACLs can do the job on the fly. You just have to set the default ACLs on the parent directory.

Further improvement may be reached by filesystem optimizations. If it is ext3/ext4 then you may run e2fsck -D from time to time. Maybe it helps to put this directory onto a separate volume. You may try different filesystems or filesystem settings (e.g. different inode sizes).

  • ACLs are good as long as you are not working on an NFSv4 mount. – ostrokach May 25 '16 at 13:22
  • The find solution about doubled my time, chmoding inside a docker container. – Nathan Arthur Dec 12 '17 at 14:55
8

Assuming the use of chmod from the GNU coreutils package on Ubuntu 12.10.

chmod 775 . -R executes the fchmodat system call for each file that it finds irrespective of whether the permissions need changing or not. I confirmed this by both inspecting the code and using strace chmod 775 . -R (snippet below) to list the actual behaviour.

newfstatat(4, "d", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "d", 0775)                  = 0
newfstatat(4, "c", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "c", 0775)                  = 0
newfstatat(4, "a", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "a", 0775)                  = 0
newfstatat(4, "b", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "b", 0775)                  = 0

There are a couple of disadvantages of running fchmodat on each file

  • The extra system call will likely become significant if a large number of files are changed. The find/xargs/chmod method mentioned by others will likely be quicker by only changing files that need changing.
  • The call to fchmodat changes the file status modification (ctime) of each file. This will cause every file/inode to change each time and will likely cause excess disk writes. It might be possible to use mount options to stop these excess writes.

A simple experiment shows the ctime changes happening for straight chmod

auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 d
auser@duncow:/tmp/blah.test$ chmod 775 . -R
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

But this does not change for find/xargs/chmod a few minutes later

auser@duncow:/tmp/blah.test$ date
Tue Jun 18 18:27:27 BST 2013
auser@duncow:/tmp/blah.test$ find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {}
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

I would always tend to use the find/xargs/chmod version because find gives more control over selecting things.

1

The [source] (1) shows, that chmod(1) always tries to set the mode and then checks back again with [fstatat(2)] (2).

Files are processed via [fts(3)] (3), which has to 'stat' all traversed file system objects beforehand to build its data tree.

Unixlore features a [nice article] (4) where chmod(1) is timed against an find / xargs approach: the latter wins by magnitudes.

Here the command line adapted to the original question:

find . -print0 | xargs -0 chmod 775

Two reasons:

  1. File system traversal is decoupled from operations on the files via the pipe between the two processes, which might even run on different cores.

    1. fts(3) operation is minimized, because xargs(1) 'flattens' out the directory tree.

So yes: you should definitely use find / xargs. for a simple solution.

Other options:

  • Play with the [umask] (5) and the source code of the process(es) writing the new files.

  • If you are using Linux, chances are your system has enabled the inotify kernel subsystem. In this case, you can script an efficient solution via [inotifywait(1)] (6).


Sidenote: unless you want execute permissions on your files, I'd suggest to modify the invocation as so:

find . -type f -print0 | xargs -0 chmod 664
find . -type d -print0 | xargs -0 chmod 775

Note to the editors: I am not allowed to add more then two links to the post, neither to comment on other posts. I leave the urls here and hope some openhearted user with sufficient reputation puts them back into the text and deletes this paragraph.


Comment on priming the disk cache with find . -printf "":

This might speed up the execution of the following chmod operations, however depends on available memory and i/o load. So it might work, or not. Decoupling traversal (find) and chmod operation already provides for caching, so priming the cache might be superfluous.

  1. https+lingrok.org/xref/coreutils/src/chmod.c#process_file
  2. https+linux.die.net/man/2/fstatat
  3. https+linux.die.net/man/3/fts
  4. http+www.unixlore.net/articles/speeding-up-bulk-file-operations.html
  5. https+en.wikipedia.org/wiki/Umask
  6. https+linux.die.net/man/1/inotifywait
0

Have you considered changing the process(es) that create the file to have them created with 0775 mode? Look at the umask value in the environment - 0002 may help.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.