29

A specific file on our production servers is being modified at apparently random times which do not appear to correlate with any log activity. We can't figure out what program is doing it, and there are many suspects. How can I find the culprit?

It is always the same file, at the same path, but on different servers and at different times. The boxes are managed by puppet, but the puppet logs show no activity at the time the file is modified.

What kernel hook, tool, or technique could help us find what process is modifying this file?

lsof is unsuitible for this, because the file is being opened, modified and closed very quickly. Any solution that relies upon polling (such as running lsof often) is no good.

  • OS: Debian testing
  • Kernels: Linux, 2.6.32 through 3.9, both 32 and 64-bit.
Improve this question
3
  • 6
    la-samhna.de/library/audit.html
    – goldilocks
    Nov 6, 2013 at 13:37
  • @goldilocks I'd give this the checkmark, if it were an answer
    – Wayne Conrad
    Nov 6, 2013 at 15:06
  • Hopefully someone comes along who can write one then, lol. I'm aware of auditd and that looks like a decent introduction, so it seemed a good suggestion -- but I haven't made use of it myself, so I'm not really qualified to elaborate.
    – goldilocks
    Nov 6, 2013 at 15:10

4 Answers 4

Reset to default
37

You can use auditd and add a rule for that file to be watched:

auditctl -w /path/to/that/file -p wa

Then watch for entries to be written to /var/log/audit/audit.log.

Improve this answer
9
  • 2
    +1 (again!). That is much better than using "lsof", which would need to catch the exact moment the modification occurs (very unlikely). You can have mode details in the top answer on this unix.se post, which mentions the use of monitoring facilities such as "auditctl", and how to set them up and use them. That is probably the safest way to ensure you know what happen(s)(ed)!
    – Olivier Dulac
    Nov 6, 2013 at 16:59
  • This was very easy to set up, and quickly found the process that was modifying the file.
    – Wayne Conrad
    Nov 7, 2013 at 12:52
  • 3
    To install: apt install auditd
    – Jonathon Reinhart
    Jan 15, 2017 at 17:51
  • how do we remove the auditctl on the fle ?
    – Siddharth
    Dec 8, 2018 at 7:45
  • 1
    Complementing this answer, auditctl -a always,exit -F dir=/path/to/that/file -F perm=wa -F uid!=<whitelisted_userid> -k mykey where it will ignore changes made my <whitelisted_userid>
    – Sandeep Kanabar
    Dec 16, 2020 at 11:07
4

SystemTap can do this, using the inodewatch script .

Improve this answer
3

in case the program(s) you are looking for still have the file opened, you can use the following:

 sudo lsof /path/to/file/being/modified

you could also call this in a small loop, suing the following script getfileusers.sh:

 #!/bin/sh

 FILE=$1

 while true; do
    lsof "${FILE}"
 done > /tmp/fileusers.log

and then call it:

 sudo ./getfileusers.sh /path/to/file/being/modified

and eventually inspect /tmp/fileusers.log to see who touched the file...

Improve this answer
1
  • I'm tempted to -1... the loop will be very heavy on CPU load (especially as it invoke "true" on each passage, and then "lsof"), and still have a great chance of missing the time the modification occurs [unless it's long enough, which is not always the case]! See StephaneChazelas answer for a much better solution, with more chance to actually catch the event(s).
    – Olivier Dulac
    Nov 6, 2013 at 17:00
2

If you can catch it in the act, that is if the file is being accessed at the moment, you can find the process by running lsof:

$ while :; do echo foo >> bar.txt; done &
$ sudo lsof bar.txt
COMMAND   PID     USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
bash    25723 terdon    1u   REG    8,6  5015796 16647496 bar.txt
Improve this answer

Not the answer you're looking for? Browse other questions tagged or ask your own question.