I won't discuss logging with regard to ubuntu specifically much, since the topic is standardized for linux in general (and I believe most or all of what I have to say is also true in general for any flavor *nix, but don't take my word for that). I also won't say much about "how to read logs" beyond answering this question:
Is the assumption even correct that they are written for human
readability or are they generally evaluated and used by other tools?
I guess that depends on the application, but in general, at least with regard to what goes into syslog (see below), they should be human readable. "Meaningful to me" is another issue, lol. However, they may be also be structured in a way that makes parsing them with standard tools (grep, awk, etc) for specific purposes easier.
Anywho, first, there is a distinction between applications which do their own logging and applications which use the system logger. Apache by default is the former, although it can be configured to do the later (which I think most people would consider undesirable). Applications which do their own logging could do so in any manner using any location for the file(s), so there is not much to say about that. The system logger is generally referred to as
"Syslog" is really a standard that is implemented with a daemon process generically called syslogd (d is for daemon!). The predominant syslog daemon currently in use on linux, including ubuntu, is
rsyslogd. Rsyslogd can do a lot, but as configured out of the box on most distros it emulates a traditional syslog, which sorts stuff into plain text files in
/var/log. Like many things *nix, rsyslogd seems to have won advocates based on functionality and not well organized documentation. Which is not to say the documentation doesn't exist -- just a lot of it might as well not, because it is scattered around in various places and finding the appropriate part is not all that straightforward. A good place to start is the stuff in
/usr/share/doc/rsyslog-doc-[version] -- except that's probably not installed (beware, there is a
/usr/share/doc/rsyslog-[version], but that's something else). To install it, you'll have to query your packaging system (eg,
apt-cache search rsyslog | grep doc).
The configuration is in
/etc/rsyslog.conf, which has a manual page,
man rsyslog.conf, although (again typically) while the manual page makes a fine reference, it looks a tad impenetrable as an introduction. Fortunately, the fundamentals of the stock rsyslog.conf conform to those of the traditional syslog.conf, for which there are various decent introductions around. This one, for example; what you want to take away from that, while peering at your local rsyslog.conf, is an understanding of facilities and priorities ("priority" is sometimes referred to as loglevel), since these are part of the aforementioned syslog standard. The reason this standard is important is because rsyslog actually gets its stuff via the kernel, and what the kernel implements is the standard.
With regard to the
$ directives in rsyslog.conf, these are rsyslog specific and if you install that optional doc package you'll find a guide to them in
Have fun ;) If you are curious about how applications use the system logger, look at
man logger and
man 3 syslog.
The normative means of rotating logs is via a tool called
logrotate (and yes, there is a
man logrotate). The normative method of using logrotate is via the cron daemon, although it does not have to be done that way (eg, if you tend to turn your desktop off everyday, you might as well just do it once at boot before syslog starts but (obviously) after the filesystem is mounted rw).
There's a good introduction to logrotate here. Note that logrotate is not just for syslog stuff, it can be used with any file at all. The base configuration file is
/etc/logrotate.conf, but since the configuration has an "include" directive, commonly most stuff goes into individual files in the
/etc/logrotate.d directory (here d is for directory, not daemon; logrotate is not a daemon).
An IMPORTANT thing to consider when using logrotate is how an application will re-act when its log file gets "rotated" -- in other words, moved -- while the application is running. WRT (r)syslogd, it will just stop writing to that log (I think there is a security justification for this). The usual way to deal with that is to tell syslog to restart (and re-open all its files), which is why you will see a
postrotate directive in logrotate conf files sending SIGHUP to the syslog daemon.