0

I have used iptables to save information about connections by modifying the /etc/syslog.conf file to save information in /var/log/iptables.log.

So, every X hours, I need to extract information from the log file. But the problem is the file will be bigger and the search will be slower. So I used:

sed -i '/string/d' iptables.log

In order to delete what I don't need anymore. But when I use this command, iptables stops saving data in the iptables.log file.

So, what's wrong? How can I solve this problem?

migrated from stackoverflow.com Nov 4 '13 at 1:48

This question came from our site for professional and enthusiast programmers.

  • 2
    Consider using logrotate. – devnull Oct 14 '13 at 9:40
1

try maybe a

cat iptables.log | sed -n "/PatternYouLookat/ p"

so you don't lock the file during the sed

on big file, sed is poor in performance so try to first grep the file or (what i do) keep a marker/index so you tail from this one and only treat the tail of the file (even with a temporary copy if treatment is heavy).

you could use the sed -u to work as a stream and avoid some buffer problem on huge file

0

I guess you're using syslog. And I'm assuming sed in-place edit actually crates a new file and removes the old one.

Syslog is very fragile if you rotate/recreate the logfiles without notifying it. What you see is exactly what happens: syslog doesn't log anything to the file even though it does exist. That is because syslog still has the old file handle and writes to the old file (even though you can't see it any more with ls etc). As soon as syslog closes this filehandle the data disappears in the nirvana.

I would reccomend like devnull to use logrotate instead. If you don't want, then issue a syslog reload or restart after your sed call. That should hopefully do the trick.

  • just to confirm: sed does recreate it. You can check that with ls -li on the logfile and you will see that the inode number has changed after the edit (first field). – gentoomaniac Oct 18 '13 at 10:48
0

Use logrotate. Normally you rotate every night or every week, but you can change this to whatever you need. The only catch is that it might mean that iptables needs to restart after rotating, which is handled by logrotate. I don't know if that is needed, nor if that (or the frequency) is a problem for you or your application.

Logrotate creates a new logfile and renames the old ones. You can keep them all, or delete them automatically after a while. The rotated file(s) can be changed without problems, because they are not used for logging anymore.

  • Say you have a logfile access.log.
  • Logrotate normally renames this to access.log-20140729.
  • Instead of a date you can use a number: access.log.1.
  • You can run another cronscript that changes access.log.1 using sed.
  • The next rotate, this file is renamed to access.log.2, with all your changes.
  • The latest access.log is renamed to access.log.1, which can then be changed by sed.
  • A new access.log is created, etc etc...

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.