Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

This is a more general question about 'chmoding' recursively.

I have this script which at some point needs to change the permissions recursively in a folder which has a few hundred thousand files. There are new files added in that folder every day, but the ones that are already there have the permissions already set and they don't change.

My question is... when I call

chmod 775 . -R 

does it try to set the permission for the files that already have the right permissions set, or only for the new files that don't have the right permissions?

It seems to always take ages to get past this command in the script, even though the 'new' files are only a few thousand and it should do their permissions fairly quickly.

I've looked at the man page for chmod, but it doesn't seem to mention anything on this case.

If chmod doesn't check for permissions beforehand, should I start looking at combining 'find' with 'chmod'?

share|improve this question
2  
I wonder if it's really slower to check the permissions and change them if they're not correct than directly set them to the correct value. –  lgeorget Jun 18 '13 at 16:46
    
if anyone stumbles upon this and wants the find + chmod command, here it is: find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {} –  Titi Dumi Jun 18 '13 at 16:46
    
@lgeorget, so you are saying it is slower to use find|chmod? than just to chmod everything. (sorry, didn't understand from your comment). cheers –  Titi Dumi Jun 18 '13 at 16:48
    
In my humble opinion, it's probably slower as it needs to lauch two processes and redirect the output of the first one to the second but I'm not sure. It depends on the time it takes to set permissions which may not be that important since they are just 3 bytes to modify in the inode. –  lgeorget Jun 18 '13 at 16:53
1  
@depquid The main performance issue here is reading the data into the disk cache. After the first run everything is in the disk cache (unless there is too little memory) thus you are testing the performance of something which is not the bottleneck in the real situation. –  Hauke Laging Jun 18 '13 at 17:25

4 Answers 4

up vote 4 down vote accepted

chmod might or might not change the permissions of files that are already set to what you want, but if not, it would still need to check them to see what their current permissions are[0]. With hundreds of thousands of files, I don't think it would matter either way; the time is most likely being spent by the tools stating every file.

You can try using find to either check for files newer than the last run or files that need chmod to be run, but I don't think you'll get much speed improvement.

If possible for your script, you might be able to get the new files put into a separate directory first, as a "holding" area. Then you can chmod THAT directory (which only has new files), and mv them in with the rest. That should be substantially faster, but unfortunately won't work for every application.

[0] Even if it does try to set the permission of files that don't need any changes, the underlying filesystem probably won't do anything with the request, because it's unnecessary.

share|improve this answer
    
Thanks for that. I will try the find | chmod version and see if it makes things faster. If not I will try to modify the script to implement a 'holding' folder as you suggested. –  Titi Dumi Jun 18 '13 at 16:56
    
The reason you wouldn't get a speed improvement is that the inode has to be read both for the ctime and the access rights. –  Hauke Laging Jun 18 '13 at 16:57

Assuming the use of chmod from the GNU coreutils package on Ubuntu 12.10.

chmod 775 . -R executes the fchmodat system call for each file that it finds irrespective of whether the permissions need changing or not. I confirmed this by both inspecting the code and using strace chmod 775 . -R (snippet below) to list the actual behaviour.

newfstatat(4, "d", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "d", 0775)                  = 0
newfstatat(4, "c", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "c", 0775)                  = 0
newfstatat(4, "a", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "a", 0775)                  = 0
newfstatat(4, "b", {st_mode=S_IFREG|0666, st_size=0, ...}, AT_SYMLINK_NOFOLLOW) = 0
fchmodat(4, "b", 0775)                  = 0

There are a couple of disadvantages of running fchmodat on each file

  • The extra system call will likely become significant if a large number of files are changed. The find/xargs/chmod method mentioned by others will likely be quicker by only changing files that need changing.
  • The call to fchmodat changes the file status modification (ctime) of each file. This will cause every file/inode to change each time and will likely cause excess disk writes. It might be possible to use mount options to stop these excess writes.

A simple experiment shows the ctime changes happening for straight chmod

auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:17 d
auser@duncow:/tmp/blah.test$ chmod 775 . -R
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

But this does not change for find/xargs/chmod a few minutes later

auser@duncow:/tmp/blah.test$ date
Tue Jun 18 18:27:27 BST 2013
auser@duncow:/tmp/blah.test$ find . ! -perm 775 -print0 | xargs -0 -I {} chmod 775 {}
auser@duncow:/tmp/blah.test$ ls -lc
total 0
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 a
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 b
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 c
-rwxrwxr-x 1 laptop laptop 0 Jun 18 18:25 d

I would always tend to use the find/xargs/chmod version because find gives more control over selecting things.

share|improve this answer

find / chmod optimization

Both find and chmod have to read

  1. all directory entries
  2. the inodes for all these entries

You probably get a performance improvement by first reading all the entries and then all the inodes (on a rotating disk) because then the disk head does not move between the directory and the inodes). As chmod is stupid (as one of the other answers explains) it should be called through find only. But even then it may help to read all the inodes before the first gets written (assuming you have enough free RAM for the disk cache). I suggest this:

find . -printf "" # reading the file names only
find . ! -perm 775 -printf "" # reading all the inodes (file names are cached)
find . ! -perm 775 -exec chmod 775 + # writing to the cache without reading from disk

The good solution: ACLs

The good solution may be completely different: If the files are created in this directory (and not moved from somewhere else) then ACLs can do the job on the fly. You just have to set the default ACLs on the parent directory.

Further improvement may be reached by filesystem optimizations. If it is ext3/ext4 then you may run e2fsck -D from time to time. Maybe it helps to put this directory onto a separate volume. You may try different filesystems or filesystem settings (e.g. different inode sizes).

share|improve this answer

Have you considered changing the process(es) that create the file to have them created with 0775 mode? Look at the umask value in the environment - 0002 may help.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.