Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Assume, that beside the Apache web server logs I never had any contact with any kind of (professional) logs on any operation system. So Logging, although I understand some basics, is all together a pretty new topic. At the moment the investment to fully learn about this topic seems to be quite huge, yet I don't even know yet, if it is even worth knowing more then the most abstract concepts.

Which resources would you suggest should someone in that situation consume (tutorials, man pages, books) to learn about Logging?

Which logs should a normal Linux user read on a daily/monthly basis? Is the assumption even correct that they are written for human readability or are they generally evaluated and used by other tools?

What should the normal *nix user and software developer know about these logs?

What do you need to know about log rotation, if you are not expected to manage professional web servers with huge loads of events?

share|improve this question
    
Do you mind if I (or you) edit the title to read "on linux" instead of "on ubuntu" since these are essentially the same question (see my answer)? That way we are less likely to end up with closed duplicate questions on "learning logging on arch", "learning logging on centOS", etc. –  goldilocks Feb 27 '13 at 15:02
    
I don't mind such kind of changes, as long as the answers are good :) –  erikb85 Mar 1 '13 at 17:09

1 Answer 1

up vote 5 down vote accepted

I won't discuss logging with regard to ubuntu specifically much, since the topic is standardized for linux in general (and I believe most or all of what I have to say is also true in general for any flavor *nix, but don't take my word for that). I also won't say much about "how to read logs" beyond answering this question:

Is the assumption even correct that they are written for human readability or are they generally evaluated and used by other tools?

I guess that depends on the application, but in general, at least with regard to what goes into syslog (see below), they should be human readable. "Meaningful to me" is another issue, lol. However, they may be also be structured in a way that makes parsing them with standard tools (grep, awk, etc) for specific purposes easier.

Anywho, first, there is a distinction between applications which do their own logging and applications which use the system logger. Apache by default is the former, although it can be configured to do the later (which I think most people would consider undesirable). Applications which do their own logging could do so in any manner using any location for the file(s), so there is not much to say about that. The system logger is generally referred to as syslog.

syslog

"Syslog" is really a standard that is implemented with a daemon process generically called syslogd (d is for daemon!). The predominant syslog daemon currently in use on linux, including ubuntu, is rsyslogd. Rsyslogd can do a lot, but as configured out of the box on most distros it emulates a traditional syslog, which sorts stuff into plain text files in /var/log. You might find documentation for it in /usr/share/doc/rsyslog-doc-[version] (beware, there is also a /usr/share/doc/rsyslog-[version], but that's just notices from the source package such as NEWS and ChangeLog). If it's there, it's html, but Stack Exchange doesn't permit embedding local file links:

file://usr/share/doc/rsyslog-doc/index.html

So you could try copy pasting that. If it's not there, it may be part of a separate package that is not installed. Query your packaging system (eg, apt-cache search rsyslog | grep doc).

The configuration is in /etc/rsyslog.conf, which has a manual page, man rsyslog.conf, although while the manual page makes a fine reference, it may be less penetrable as an introduction. Fortunately, the fundamentals of the stock rsyslog.conf conform to those of the traditional syslog.conf, for which there are many introductions and tutorials around. This one, for example; what you want to take away from that, while peering at your local rsyslog.conf, is an understanding of facilities and priorities ("priority" is sometimes referred to as loglevel), since these are part of the aforementioned syslog standard. The reason this standard is important is because rsyslog actually gets its stuff via the kernel, and what the kernel implements is the standard.

With regard to the $ directives in rsyslog.conf, these are rsyslog specific and if you install that optional doc package you'll find a guide to them in rsyslog_conf_global.html.

Have fun...if you are curious about how applications use the system logger, look at man logger and man 3 syslog.

Log Rotation

The normative means of rotating logs is via a tool called logrotate (and there is a man logrotate). The normative method of using logrotate is via the cron daemon, although it does not have to be done that way (e.g., if you tend to turn your desktop off everyday, you might as well just do it once at boot before syslog starts but, obviously, after the filesystem is mounted rw).

There's a good introduction to logrotate here. Note that logrotate is not just for syslog stuff, it can be used with any file at all. The base configuration file is /etc/logrotate.conf, but since the configuration has an "include" directive, commonly most stuff goes into individual files in the /etc/logrotate.d directory (here d is for directory, not daemon; logrotate is not a daemon).

An IMPORTANT thing to consider when using logrotate is how an application will re-act when its log file gets "rotated" -- in other words, moved -- while the application is running. WRT (r)syslogd, it will just stop writing to that log (I think there is a security justification for this). The usual way to deal with that is to tell syslog to restart (and re-open all its files), which is why you will see a postrotate directive in logrotate conf files sending SIGHUP to the syslog daemon.

For applications that do not implement this but like to hold a log handle open, I have a simple intermediate application, pipelog, using a similar system to rotate on a signal.

share|improve this answer
1  
If you mention syslog-ng, too, you will have written all there is to say on linux-logging. Excellent answer. –  Nils Feb 27 '13 at 20:28
    
I learned a lot and have more sources to read. Thanks! Btw. Why would it be bad to use system logging for apache? And which logs should I want to read? –  erikb85 Mar 1 '13 at 18:16
    
@erikb85 1) Mostly because apache generates a lot of output. I have not had to configure apache in a long time, so I can't remember how flexible it is WRT to sending specific things to syslog (but I'd guess very flexible) 2) WRT logs you want to read, have a look at the logic in your /etc/rsyslog.conf. For example: often stuff above a certain priority will go into /var/log/messages, and stuff below that will go into /var/log/notice. I like to also have a log that contains everything, which makes for double and triple overlap, but if you keep them rotated it is no big deal. –  goldilocks Mar 2 '13 at 16:22
    
Just to confirm, rsyslog is not a daemon wrapper around logrotate, right ? Why so ? –  Thomas Nov 20 at 10:54
    
@Thomas They perform different purposes, although (r)syslog could implement rotation to spare the need for logrotate. However, having them separate means you can use logrotate on things other than syslog files. –  goldilocks Nov 20 at 13:04

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.