Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

Assume, that beside the Apache web server logs I never had any contact with any kind of (professional) logs on any operation system. So Logging, although I understand some basics, is all together a pretty new topic. At the moment the investment to fully learn about this topic seems to be quite huge, yet I don't even know yet, if it is even worth knowing more then the most abstract concepts.

Which resources would you suggest should someone in that situation consume (tutorials, man pages, books) to learn about Logging?

Which logs should a normal Linux user read on a daily/monthly basis? Is the assumption even correct that they are written for human readability or are they generally evaluated and used by other tools?

What should the normal *nix user and software developer know about these logs?

What do you need to know about log rotation, if you are not expected to manage professional web servers with huge loads of events?

share|improve this question
    
Do you mind if I (or you) edit the title to read "on linux" instead of "on ubuntu" since these are essentially the same question (see my answer)? That way we are less likely to end up with closed duplicate questions on "learning logging on arch", "learning logging on centOS", etc. –  TAFKA 'goldilocks' Feb 27 '13 at 15:02
    
I don't mind such kind of changes, as long as the answers are good :) –  erikb85 Mar 1 '13 at 17:09
add comment

1 Answer

up vote 3 down vote accepted

I won't discuss logging with regard to ubuntu specifically much, since the topic is standardized for linux in general (and I believe most or all of what I have to say is also true in general for any flavor *nix, but don't take my word for that). I also won't say much about "how to read logs" beyond answering this question:

Is the assumption even correct that they are written for human readability or are they generally evaluated and used by other tools?

I guess that depends on the application, but in general, at least with regard to what goes into syslog (see below), they should be human readable. "Meaningful to me" is another issue, lol. However, they may be also be structured in a way that makes parsing them with standard tools (grep, awk, etc) for specific purposes easier.

Anywho, first, there is a distinction between applications which do their own logging and applications which use the system logger. Apache by default is the former, although it can be configured to do the later (which I think most people would consider undesirable). Applications which do their own logging could do so in any manner using any location for the file(s), so there is not much to say about that. The system logger is generally referred to as syslog.

syslog

"Syslog" is really a standard that is implemented with a daemon process generically called syslogd (d is for daemon!). The predominant syslog daemon currently in use on linux, including ubuntu, is rsyslogd. Rsyslogd can do a lot, but as configured out of the box on most distros it emulates a traditional syslog, which sorts stuff into plain text files in /var/log. Like many things *nix, rsyslogd seems to have won advocates based on functionality and not well organized documentation. Which is not to say the documentation doesn't exist -- just a lot of it might as well not, because it is scattered around in various places and finding the appropriate part is not all that straightforward. A good place to start is the stuff in /usr/share/doc/rsyslog-doc-[version] -- except that's probably not installed (beware, there is a /usr/share/doc/rsyslog-[version], but that's something else). To install it, you'll have to query your packaging system (eg, apt-cache search rsyslog | grep doc).

The configuration is in /etc/rsyslog.conf, which has a manual page, man rsyslog.conf, although (again typically) while the manual page makes a fine reference, it looks a tad impenetrable as an introduction. Fortunately, the fundamentals of the stock rsyslog.conf conform to those of the traditional syslog.conf, for which there are various decent introductions around. This one, for example; what you want to take away from that, while peering at your local rsyslog.conf, is an understanding of facilities and priorities ("priority" is sometimes referred to as loglevel), since these are part of the aforementioned syslog standard. The reason this standard is important is because rsyslog actually gets its stuff via the kernel, and what the kernel implements is the standard.

With regard to the $ directives in rsyslog.conf, these are rsyslog specific and if you install that optional doc package you'll find a guide to them in /usr/share/doc/rsyslog-doc-7.2.5/rsyslog_conf_global.html.

Have fun ;) If you are curious about how applications use the system logger, look at man logger and man 3 syslog.

Log Rotation

The normative means of rotating logs is via a tool called logrotate (and yes, there is a man logrotate). The normative method of using logrotate is via the cron daemon, although it does not have to be done that way (eg, if you tend to turn your desktop off everyday, you might as well just do it once at boot before syslog starts but (obviously) after the filesystem is mounted rw).

There's a good introduction to logrotate here. Note that logrotate is not just for syslog stuff, it can be used with any file at all. The base configuration file is /etc/logrotate.conf, but since the configuration has an "include" directive, commonly most stuff goes into individual files in the /etc/logrotate.d directory (here d is for directory, not daemon; logrotate is not a daemon).

An IMPORTANT thing to consider when using logrotate is how an application will re-act when its log file gets "rotated" -- in other words, moved -- while the application is running. WRT (r)syslogd, it will just stop writing to that log (I think there is a security justification for this). The usual way to deal with that is to tell syslog to restart (and re-open all its files), which is why you will see a postrotate directive in logrotate conf files sending SIGHUP to the syslog daemon.

share|improve this answer
1  
If you mention syslog-ng, too, you will have written all there is to say on linux-logging. Excellent answer. –  Nils Feb 27 '13 at 20:28
    
I learned a lot and have more sources to read. Thanks! Btw. Why would it be bad to use system logging for apache? And which logs should I want to read? –  erikb85 Mar 1 '13 at 18:16
    
@erikb85 1) Mostly because apache generates a lot of output. I have not had to configure apache in a long time, so I can't remember how flexible it is WRT to sending specific things to syslog (but I'd guess very flexible) 2) WRT logs you want to read, have a look at the logic in your /etc/rsyslog.conf. For example: often stuff above a certain priority will go into /var/log/messages, and stuff below that will go into /var/log/notice. I like to also have a log that contains everything, which makes for double and triple overlap, but if you keep them rotated it is no big deal. –  TAFKA 'goldilocks' Mar 2 '13 at 16:22
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.