5

I have a Solaris 10 server with autofs-mounted home dirs. On one server they are not unmounted after the 10 min timeout period. We've got AUTOMOUNT_TIMEOUT=600 in /etc/default/autofs, I ran automount -t 600, disabled and re-enabled svc:/system/filesystem/autofs:default service and nothing seems to work.

My suspicion is that something on the system is periodically accessing all the mounted filesystems, maybe checking if they are accessible, and thus resetting the automounter timeout that in turn never expires. This is supported by a test I just did - if I set the timeout to 10 seconds the mountpoints are unmounted, looks like 10 sec is shorter than the period in which that something is doing the checks and the timer has a chance to expire.

The question is how can I find what process is doing that? The server is a heavily used production system and I can't do any dangerous experiments on it.

Note that the filesystems are not kept open and can be manually unmounted. That something is probably going mountpoint by mountpoint, cd in, cd out, move on, often enough to prevent automount from unmounting it. But it doesn't keep it open and therefore is not visible with lsof or fuser -c. I want to catch it or record it as soon as it accesses the mountpoints to know what's doing it.

FWIW it's a Solaris 10 zone on rather beefy Solaris 10 host (Sparc / M5000).

4

You could try rwsnoop (http://dtracebook.com/index.php/File_System:rwsnoop) to monitor i/o access using dtrace:

# rwsnoop - snoop read/write events.
#           Written using DTrace (Solaris 10 3/05).
#
# This is measuring reads and writes at the application level. This matches
# the syscalls read, write, pread and pwrite.

good luck!

  • That looks promising, thanks! I don't want to filter by PID, instead by mountpoint, but will take that script as a starting point. Thanks again! – MLu Jul 11 '13 at 9:10
  • the script outputs the path, so you can always use grep to filter the data. – mrc Jul 11 '13 at 10:44
2

You can use the tool lsof to determine what process is accessing a particular file on the filesystem. In this case you can use it to detect what process is accessing the automount's mount point.

Example

I have the following automount for my /vz_backups directory:

$ showmount -e krycek
Export list for krycek:
/export/raid1/vz_backups 192.168.1.0/24

Now cd to an automount under the mount point:

$ cd /vz_backups/images

See that we've mounted it:

$ mount | grep /vz
krycek:/export/raid1/vz_backups/images on /vz_backups/images type nfs (rw,intr,tcp,rsize=16384,wsize=16384,addr=192.168.1.4)

You can then interrogate file accesses using lsof:

$ lsof -l | grep /vz
automount  3359        0   16r      DIR       0,24           0      11960 /vz_backups
bash      28635        0  cwd       DIR       0,30        4096    2981890 /vz_backups/images (krycek:/export/raid1/vz_backups/images)
lsof      31040        0  cwd       DIR       0,30        4096    2981890 /vz_backups/images (krycek:/export/raid1/vz_backups/images)
grep      31041        0  cwd       DIR       0,30        4096    2981890 /vz_backups/images (krycek:/export/raid1/vz_backups/images)
lsof      31042        0  cwd       DIR       0,30        4096    2981890 /vz_backups/images (krycek:/export/raid1/vz_backups/images)

Now cd out of that directory so that nothing is accessing it, and either wait for the automount to timeout, or manually umount it and then check it again:

$ cd ~
$ umount /vz_backups/images

$ lsof -l | grep /vz
automount  3359        0   16r      DIR       0,24           0      11960 /vz_backups

References

  • Thanks for that but it's not that simple. Trouble is there is no process that has the mountpoint permanently open (therefore lsof or even simpler fuser -c /mount/point/ comes out empty). I suspect that from time to time some process goes bang bang gang one filesystem after another, cd in, cd out, move on. Therefore I need to catch or record it when it does so. I don't know what is it nor how often it accesses the mountpoints. That's what I want to find out. – MLu Jul 11 '13 at 4:55
0

You probably have monitoring software running which runs the 'df' command in an interval lower than your automounter timeout. On Solaris, the statvfs system call marks a filesystem as in use. Solution : use df -l.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.