No output from locate command

Question

I usually find the answers to all my Unix related problems already posted as questions and answers. However, this particular issue has had me stumped for the past hour so I thought I’d ask my first question on this site.

Problem

I have a development / staging server server running CentOS 5.11. Running locate as a regular user results in no output (not even an error message):

locate readdir

However, running the command as the superuser prints a list of valid results:

$ sudo locate readdir
/home/anthony/repos/php-src/TSRM/readdir.h
/home/anthony/repos/php-src/ext/standard/tests/dir/readdir_basic.phpt
... etc.

strace usually helps me debug any such issues and running strace locate readdir shows:

stat64("/var/lib/mlocate/mlocate.db", 0xbff65398) = -1 EACCES (Permission denied)
access("/", R_OK|X_OK)                  = -1 EACCES (Permission denied)
exit_group(1)                           = ?

Check permissions

I checked the ownership and permissions of the locate binary and its default database. As expected the command is setgid with slocate as the group owner while the database has the appropriate ownership and permissions.

$ ls -l /usr/bin/locate
-rwx--s--x 1 root slocate 22280 Sep  3  2009 /usr/bin/locate

$ sudo ls -l /var/lib/mlocate/mlocate.db
-rw-r----- 1 root slocate 78395703 May  8 04:02 /var/lib/mlocate/mlocate.db

$ sudo ls -ld /var/lib/mlocate/
drwxr-x--- 2 root slocate 4096 Sep  3  2009 /var/lib/mlocate/

There are also no unusual file attributes:

$ sudo lsattr /usr/bin/locate /var/lib/mlocate/mlocate.db
------------- /usr/bin/locate
------------- /var/lib/mlocate/mlocate.db

Compare with working system

Meanwhile, everything works as expected on the Production server. Running locate readdir as a regular (non-root) user returns a list of results as it should:

$ locate readdir
/usr/include/php/TSRM/readdir.h
/usr/lib/perl5/5.8.8/i386-linux-thread-multi/auto/POSIX/readdir.al
/usr/share/man/man2/readdir.2.gz

For comparison, I also ran this command through strace but I then got the same permission denied error as on the staging server. I was wondering how this could be until I read the manual page for sudo. Listed in the Bugs section:

Programs that use the setuid bit do not have effective user ID privileges while being traced.

So, unfortunately, I can’t use strace for debugging.

I compared the results of all the above commands between the Staging and Production servers and there’s no difference between them. Both systems have the mlocate-0.15-1.el5.2 RPM with no modifications to their files as shown by rpm -V mlocate.

Other considerations

I thought it might be related to the fact that on the problematic staging server, my login is authenticated using Winbind but I created a regular local user on the same box and I still have the same issue. There’s obviously something else that I’m missing but I simply don’t know what it is.

I suspect it is related to the setgid file permission, maybe PAM or possibly SELinux. I don’t know much about either PAM or SELinux: I’ve only ever looked at PAM when configuring Winbind authentication while SELinux was installed with the OS but I’ve never used it.

Note: the production server has been subject to far fewer modifications than the development server which has had some experimentation.

Answer 1

The problem was the permissions for / (the root directory) and the clue for finding that was this line from your strace output:

access("/", R_OK|X_OK)                  = -1 EACCES (Permission denied)

You were missing group read permission settings for /. But because you still had x (execute) permission, which allows you to traverse a directory, you could still access all of the files on the filesystem, which is why most everything continued working while those permissions were in effect. The only thing you were not allowed to do is list the contents of /. Most commands don't need to list /, they either use pathnames relative to the current directory or absolute pathnames that access specific well-known directories off the root (like /etc and /var).

For security reasons, locate, even though it has access to a complete inventory of filenames generated by a privileged user, insists on reporting only results that the calling user would be able to find by scanning the whole filesystem from the root. Since you couldn't list /, which makes scanning anything straight from the root a non-starter, locate would report nothing at all.

Answer 2

Thanks to Celada’s keen eye and attention to detail, I checked the permissions of the root directory.

$ sudo ls -ld /
drwx--xr-x 25 root domain users 4096 Apr 22 17:57 /

When I authenticate using my Active Directory credentials, domain users is the primary group allocated to me by Winbind. A while ago I had been denying read privileges on a set of directories to other users by changing group ownership and removing read permission from the group. I must have accidentally done the same to the root directory. Doh! Interestingly, that was a few weeks ago and I hadn’t noticed any problems - though it coincided with the time where I consciously decided to start logging in as a regular user and using sudo rather than logging in as root.

Anyhow, I changed the permissions back to what they should be:

$ sudo chgrp root /
$ sudo chmod -v g+r /
mode of `/' changed to 0755 (rwxr-xr-x)

$ ls -ld /
drwxr-xr-x 25 root root 4096 Apr 22 17:57 /

Now locate works as it should.