25

A specific file on our production servers is being modified at apparently random times which do not appear to correlate with any log activity. We can't figure out what program is doing it, and there are many suspects. How can I find the culprit?

It is always the same file, at the same path, but on different servers and at different times. The boxes are managed by puppet, but the puppet logs show no activity at the time the file is modified.

What kernel hook, tool, or technique could help us find what process is modifying this file?


lsof is unsuitible for this, because the file is being opened, modified and closed very quickly. Any solution that relies upon polling (such as running lsof often) is no good.


  • OS: Debian testing
  • Kernels: Linux, 2.6.32 through 3.9, both 32 and 64-bit.
3
  • 6
  • @goldilocks I'd give this the checkmark, if it were an answer – Wayne Conrad Nov 6 '13 at 15:06
  • Hopefully someone comes along who can write one then, lol. I'm aware of auditd and that looks like a decent introduction, so it seemed a good suggestion -- but I haven't made use of it myself, so I'm not really qualified to elaborate. – goldilocks Nov 6 '13 at 15:10
31

You can use auditd and add a rule for that file to be watched:

auditctl -w /path/to/that/file -p wa

Then watch for entries to be written to /var/log/audit/audit.log.

9
  • 2
    +1 (again!). That is much better than using "lsof", which would need to catch the exact moment the modification occurs (very unlikely). You can have mode details in the top answer on this unix.se post, which mentions the use of monitoring facilities such as "auditctl", and how to set them up and use them. That is probably the safest way to ensure you know what happen(s)(ed)! – Olivier Dulac Nov 6 '13 at 16:59
  • This was very easy to set up, and quickly found the process that was modifying the file. – Wayne Conrad Nov 7 '13 at 12:52
  • 2
    To install: apt install auditd – Jonathon Reinhart Jan 15 '17 at 17:51
  • how do we remove the auditctl on the fle ? – Siddharth Dec 8 '18 at 7:45
  • To install on CentOS: yum install audit – alphaGeek Aug 29 '19 at 8:47
4

SystemTap can do this, using the inodewatch script .

3

in case the program(s) you are looking for still have the file opened, you can use the following:

 sudo lsof /path/to/file/being/modified

you could also call this in a small loop, suing the following script getfileusers.sh:

 #!/bin/sh

 FILE=$1

 while true; do
    lsof "${FILE}"
 done > /tmp/fileusers.log

and then call it:

 sudo ./getfileusers.sh /path/to/file/being/modified

and eventually inspect /tmp/fileusers.log to see who touched the file...

1
  • I'm tempted to -1... the loop will be very heavy on CPU load (especially as it invoke "true" on each passage, and then "lsof"), and still have a great chance of missing the time the modification occurs [unless it's long enough, which is not always the case]! See StephaneChazelas answer for a much better solution, with more chance to actually catch the event(s). – Olivier Dulac Nov 6 '13 at 17:00
2

If you can catch it in the act, that is if the file is being accessed at the moment, you can find the process by running lsof:

$ while :; do echo foo >> bar.txt; done &
$ sudo lsof bar.txt
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    25723 terdon    1u   REG    8,6  5015796 16647496 bar.txt

Not the answer you're looking for? Browse other questions tagged or ask your own question.