I won't discuss logging with regard to ubuntu specifically much, since the topic is standardized for linux in general (and I believe most or all of what I have to say is also true in general for any flavor *nix, but don't take my word for that). I also won't say much about "how to read logs" beyond answering this question:
Is the assumption even correct that they are written for human
readability or are they generally evaluated and used by other tools?
I guess that depends on the application, but in general, at least with regard to what goes into syslog (see below), they should be human readable. "Meaningful to me" is another issue, lol. However, they may be also be structured in a way that makes parsing them with standard tools (grep, awk, etc) for specific purposes easier.
Anywho, first, there is a distinction between applications which do their own logging and applications which use the system logger. Apache by default is the former, although it can be configured to do the later (which I think most people would consider undesirable). Applications which do their own logging could do so in any manner using any location for the file(s), so there is not much to say about that. The system logger is generally referred to as
"Syslog" is really a standard that is implemented with a daemon process generically called syslogd (d is for daemon!). The predominant syslog daemon currently in use on linux, including ubuntu, is
rsyslogd. Rsyslogd can do a lot, but as configured out of the box on most distros it emulates a traditional syslog, which sorts stuff into plain text files in
/var/log. You might find documentation for it in
/usr/share/doc/rsyslog-doc-[version] (beware, there is also a
/usr/share/doc/rsyslog-[version], but that's just notices from the source package such as
ChangeLog). If it's there, it's html, but Stack Exchange doesn't permit embedding local file links:
So you could try copy pasting that. If it's not there, it may be part of a separate package that is not installed. Query your packaging system (eg,
apt-cache search rsyslog | grep doc).
The configuration is in
/etc/rsyslog.conf, which has a manual page,
man rsyslog.conf, although while the manual page makes a fine reference, it may be less penetrable as an introduction. Fortunately, the fundamentals of the stock rsyslog.conf conform to those of the traditional syslog.conf, for which there are many introductions and tutorials around. This one, for example; what you want to take away from that, while peering at your local rsyslog.conf, is an understanding of facilities and priorities ("priority" is sometimes referred to as loglevel), since these are part of the aforementioned syslog standard. The reason this standard is important is because rsyslog actually gets its stuff via the kernel, and what the kernel implements is the standard.
With regard to the
$ directives in rsyslog.conf, these are rsyslog specific and if you install that optional doc package you'll find a guide to them in
Have fun...if you are curious about how applications use the system logger, look at
man logger and
man 3 syslog.
The normative means of rotating logs is via a tool called
logrotate (and there is a
man logrotate). The normative method of using logrotate is via the cron daemon, although it does not have to be done that way (e.g., if you tend to turn your desktop off everyday, you might as well just do it once at boot before syslog starts but, obviously, after the filesystem is mounted rw).
There's a good introduction to logrotate here. Note that logrotate is not just for syslog stuff, it can be used with any file at all. The base configuration file is
/etc/logrotate.conf, but since the configuration has an "include" directive, commonly most stuff goes into individual files in the
/etc/logrotate.d directory (here d is for directory, not daemon; logrotate is not a daemon).
An IMPORTANT thing to consider when using logrotate is how an application will re-act when its log file gets "rotated" -- in other words, moved -- while the application is running. WRT (r)syslogd, it will just stop writing to that log (I think there is a security justification for this). The usual way to deal with that is to tell syslog to restart (and re-open all its files), which is why you will see a
postrotate directive in logrotate conf files sending SIGHUP to the syslog daemon.
For applications that do not implement this but like to hold a log handle open, I have a simple intermediate application, pipelog, using a similar system to rotate on a signal.