Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

On one production linux box, I am capturing SIP packets using the following configuration:

logrotate.conf:

# Opensips SIP traces 
/var/log/sip.log
{
        rotate 31
        daily
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        postrotate
                pkill tcpdump
                /home/ubuntu/log-sip-messages.sh &
        endscript
}

log-sip-messages.sh:

#!/bin/sh
tcpdump host 159.63.X.X -s0 -v >> /var/log/sip-159.63.X.X.log

This works great on the Ubuntu 11.04 server, but on the Ubuntu 12.10 boxes, this just produces a file with a timestamp at the time the logs were rotated. It's as if the pkill doesn't complete before running tcpdump again to start writing to the new file.

I've tried running this directly from the terminal as an experiment:

pkill tcpdump && tcpdump -s0 -v udp >> /var/log/sip.log

and then when I run ps aux|grep tcpdump there's no process running, confirming what I already know... the pkill command runs asynchronously on 12.10 (but either is synchronous on 11.04 or just beats the 2nd command)

How can I make this work so I can capture network traffic in a nice, readable log file (I don't need pcaps) without risking filling up my hard drive space?

share|improve this question
    
As an aside, do we need a pkill tag or is kill enough to convey the message? –  jmort253 Aug 29 '13 at 23:49
2  
We haven't felt the need for a pkill tag in three years, and this question isn't related to pkill specifically but to signals in general, so I don't see the point of adding a pkill tag. –  Gilles Aug 29 '13 at 23:54
add comment

3 Answers

Unix signal delivery is asynchronous. When the kill system call returns, the signal has been delivered to the process, but the process may not have reacted to it yet. You were lucky with the scheduler under 11.04. If the killed process has a handler for the signal, it can spend an arbitrarily long time before dying, or choose not to die as a result of the signal.

You could use a lock to ensure that the new instance of the process doesn't start until the previous one has finished.

Furthermore, I don't recommend indiscriminately killing tcpdump processes: what if there are other instances running? Instead, kill any process that has the lock file open.

#!/bin/sh
lockfile=/var/run/log-sip-messages.lock
# Kill all processes that have the lock file open
fuser -k -TERM "$lockfile" >/dev/null 2>/dev/null
(
  # Wait until the lock is released
  flock -s 3
  # Don't let this shell be killed by fuser: wait until tcpdump exits
  trap : TERM
  # Call tcpdump with the lock file open, so that fuser kills it
  tcpdump -s0 -v udp >> /var/log/sip.log
) 3>"$lockfile"

And from logrotate, run /home/ubuntu/log-sip-messages.sh only.

share|improve this answer
    
do you mind if I expand how this works? –  slm Aug 30 '13 at 0:12
    
Yeh, killing tcpdump was sort of messy and inelegant. It's a start, but I definitely like the lock file idea. @slm, I personally wouldn't mind expansion on how it works. –  jmort253 Aug 30 '13 at 0:26
    
@jmort253 I've added comments, are they enough? And I also fixed a race condition. –  Gilles Aug 30 '13 at 0:28
    
@Gilles - yeah that's much clearer now, I know it makes it look busier now, but it's easier to understand. –  slm Aug 30 '13 at 0:30
    
If you want to read more about how to use trap check out this guide: linuxcommand.org/wss0160.php –  slm Aug 30 '13 at 0:46
add comment

One idea would be to introduce some wait time between the pkill and the starting back up of the tcpdump (/home/ubuntu/log-sip-messages.sh &). This feels like a hack, and it is, but something like this:

    postrotate
            pkill tcpdump
            sleep 3
            /home/ubuntu/log-sip-messages.sh &
    endscript
share|improve this answer
add comment

All of this inspired me to dig into the logrotate manpage to see what sort of tools are at my disposal. I discovered that copytruncate makes a copy of the log file and then truncates it, but without disrupting the process that writes to it. My testing shows that it does rotate the logs and allow tcpdump to continue writing to the cleared file. I left an hourly cronjob running overnight using a test configuration file so I would rotate these logs every hour. It worked:

# Opensips SIP traces 
/var/log/sip.log
{
        rotate 31
        daily
        missingok
        notifempty
        compress
        delaycompress
        sharedscripts
        copytruncate
}

One possible downside is that, if the tcpdump process is killed by someone who doesn't know what I'm doing, I'd have to come in and manually restart the process, but this is sufficient for now.

The other potential issue is that, if the log file is very large, making a copy may briefly require a lot of disk space, for which I have plenty at the moment, but the process may be interrupted if there isn't enough room to make a copy.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.