Why are there so many log files in a typical Linux system? Why don't they use one consolidated logs db/file and an api?

Question

I just wonder why are there so many log files in a typical Linux system? Wouldn't it be a better idea to have one system api function for logging and one consolidated table to save all the log entries from all the applications?

Answer 1

It's part of the Unix philosophy. The idea is that text files are free from program lock-in and everyone can use whatever technique they prefer. To take this further, flat files are often used, as opposed to markup languages like XML (although I have seen programs storing things in XML format as well).

In googling I have found this nice writeup about plain text, with comments about the Unix philosophy.

Answer 2

Using simple text files has the advantage that you don't need any database specific tools to get your log entries.

You can analyze them with grep if you like, you can open them up with your favorite pager and you can process them in your favorite scripting language like Perl, Python, etc. without the need of any additional libraries.

On a Unix system you have already some kind of "system log API". It is called syslog. Syslog is not really an API but it is an standard for logging messages. The name stands for the networking protocol and the library and daemon behind it.

The default configuration of the most systems is a syslog daemon listening to local messages.

The daemon accepts the messages and does the logging. There are several different implementations of syslog daemons for all kind of platforms and it is also possible to log your messages to a database.

It is up to you.

Answer 3

I just wonder why are there so many log files in a typical Linux system?

The different log files contain different information (though there is typically some duplication). They often have different characteristics: different rotation and retention policies, different permissions, etc. The syslog daemon takes care of writing them; you can see its settings in /etc/syslog.conf or /etc/syslog-ng.conf.

Wouldn't it be a better idea to have one system api function for logging

This one is a good idea. Let's call it syslog. Its job is to send the log entries to the syslog daemon.

and one consolidated table to save all the log entries from all the applications?

Now that's a whole can of worms. You seem to be assuming the presence of a database engine, probably a relational database, probably one you can query in SQL. But Unix is older than SQL, and there are very good reasons why it hasn't adopted SQL as a standard component. Under Unix, the database is the filesystem. It's not a relational database, it's a simple one. Its entries are not rows, but simple files, preferably text, preferably with a simple format. For example, log files are text files, with one entry per line, containing the date, the machine name, the originating program and the entry text. Using a relational database would have a number of downsides:

• What do you do if the database isn't working? (The filesystem is a fundamental component (and have I mentioned it's a lot simpler than a relational database?); the syslog daemon is a simple component that does one job (a common feature in Unix design) and so is expected to do it well and reliably.)
• How do you log database operations? (Ok, through the database itself — after all the logs contain entries from the kernel and from the syslog daemon — but again a much more complex database makes this more difficult and less reliable).
• How do you access log entries? Compare the simplicity of cat, grep, less against SQL queries. And file permissions against, well, I don't know how you'd handle this in a typical relational database.
• Multi-server installations don't store their logs locally, they use the remote log feature that's been built into the syslog daemon since pretty much the dawn of Unix. That's easy to implement with the unix logging architecture; you can't run a replicated database on that complexity budget.

Answer 4

If you would really like to store your system logs in a relational database (which can have many advantages), check out rsyslog (The up and coming replacement for syslog), which can write system logs to a MySQL, Postgres or Oracle database.

Answer 5

This would make things like 'tail -f /var/log/apache/access.log' impossible.

Why do you think it would be better to put everything in one file?