4

We're moving websites from one server configuration to a new configuration and the websites will live in different paths than previously. We're planning to diligently go through and replace old paths with new paths, but in case we miss any, is there some way to monitor for any processes trying to access the old paths and also know what UID the process was owned by?

5
  • What's your OS? Chances are the best way is to configure auditing for your OS to record failed attempts to open files. – Andrew Henle Jul 24 '15 at 13:20
  • @AndrewHenle CloudLinux 6 - so effectively CentOS 6. – sa289 Jul 24 '15 at 16:18
  • ------------------strace – PSkocik Jul 27 '15 at 23:49
  • @PSkocik would that be the best way to do it? If there are dozens to hundreds of Apache and PHP processes running and all were being straced at once, would that add a lot of overhead? I don't know enough to say it would, but that was my initial reaction. Thoughts? – sa289 Jul 27 '15 at 23:52
  • It would have some overhead, the slowdown shouldn't be that big, though. It should be proportional to the number of syscall your application makes. You can limit by limiting yourself to a subset of syscalls with "-e trace=..." switch. Another alternative would be to emulate what tup does in regards to tracing IO, which is set up a fuse file system that records the operations. Strace should be easier to set up. – PSkocik Jul 27 '15 at 23:57
6
+50

You can use this little systemtap script :

#!/usr/bin/stap
function proc:string() { return sprintf("PID(%d) UID(%d) PROC(%s)", pid(), uid(), execname()) }

probe syscall.open.return, syscall.stat.return,
  syscall.open64.return ?, syscall.stat64.return ? {
  filename = user_string($filename)
  if ($return < 0) {
    printf("failed %s on %s by %s\n", pn(), proc(), filename)
  }
}

It will hook the syscalls open and stat (you can copy/paste the code, maybe I forgot some other syscalls) at the return. As syscalls are the only way to communicate with the kernel, you won't miss anything. This script will produce this kind of output :

failed syscall.stat.return on PID(4203) UID(1000) PROC(bash) by /tmp/rofl
failed syscall.stat.return on PID(4203) UID(1000) PROC(bash) by /tmp/hihi

among the pros of using systemtap, we have :

  • less instrusive for the process
  • system-wide (not only the monitored process) but you can reduce its selection directly in the script
  • less ressource hungry (only displays failed actions, not all to be grep after)
  • you can improve the script to get the details about the calling program (eg, its backtrace, time of call, etc...). It depends on your application.

And for the cons :

  • not standard, you have to install it (but standard enough to be available on most distribution). On Redhat & variants: sudo yum install systemtap
  • need to have the debuginfos to build the module. On Redhat & variants : sudo debuginfo-install kernel

Some useful links : The tapset (included functions) index, and a beginners guide

Good luck for your migration !

1

Something like this should do it:

strace -f \
-e trace=open,stat,stat64,lstat,lstat64,chdir,mkdir,rename,symlink,creat \
 -o >(grep "the paths you want to catch" > log) \
commandToStartYourServer

You want the -f switch to track children processes. The trace options are a subset of what fabricate uses to trace IO (fabricate traces "open,stat,stat64,lstat,lstat64,execve,exit_group,chdir,mkdir,rename,clone,vfork,fork,symlink,creat")

This further limits disk IO by filtering the output via grep and process substitution (basically a pipe at the system level).

1

You could use fatrace. Kernel support (fanotify) should be present in most modern systems, but the fatrace userspace app is not in the standard repositories for some OSes. fatrace is particularly nice if your old files are on a separate file system.

http://www.lanedo.com/filesystem-monitoring-linux-kernel/

--

Far more common, though a little harder to use is auditd.

http://linux-audit.com/configuring-and-auditing-linux-systems-with-audit-daemon/

--

There is a fundamental problem with monitoring file usage by path when symlinks allow access via completely different paths, so you might need to pay special attention to that.

3
  • Can fanotify report accesses to non-existent files / dirs? From that page you linked to it sounds like it works by operating on open file descriptors which I'm thinking wouldn't exist for a non-existent file, but maybe I'm misunderstanding. Either way it's interesting to know about though. Unfortunately I'm on 2.6.32 kernel anyways b/c CentOS 6 =(. – sa289 Jul 31 '15 at 16:38
  • actually, no, probably not. – mc0e Jul 31 '15 at 18:44
  • 1
    fanotify doesn't appear to report on attempted access to non-existent files. Something I sometimes do when I'm not sure if it's OK to delete something is to first chmod 0 it to block all access. It's nice in that it's more reversable if it turns out the file is needed, but also it might help here. inotify should work on your kernel, and there are various things built on it like fam. inotify requires registering all the directories you want to watch, which might be large in your case, so something that watches all kernel events like auditd or systemtap might be more appropriate. – mc0e Jul 31 '15 at 18:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.