how are early logs logged in rsyslog, when rsyslog is not yet running?

Question

I am using rsyslog on Debian Buster.

I am using old-style sysvinit, not systemd. rsyslog is started late in the init startup sequence, after most init scripts have run.

The dmesg messages during boot process and other early log messages from other startup scripts that ran before rsyslog was started, are not lost. rsyslog will log them immediately when it starts.

Where were these messages stored, when rsyslog was not running ?

Can I clear this "cache" before I start rsyslog, so that the early boot message are not logged ?

Or, even better, can I filter them out, so that they are logged into main syslog file:

*.*    -/var/log/syslog

but not logged into another:

*.*    |/dev/xconsole

UPDATE AND CLARIFICATION

I realized it might not be entirely clear what problem I am trying to solve. So here is some clarification of the situation:

In addition to standard syslog logging (ie, /var/log/syslog), I am also logging the same info to /dev/xconsole, and when I am logged in my desktop session as user, I have terminal on background with following command:

cat /dev/xconsole

that way, I can immediately see new logs appearing in my background. Also, unlike regular file, /dev/xconsole is emptied after it has been read. So when i log out, and log back in again, I don't see old messages, but only new ones.

Now, the problem is that after boot, there is so much logs in the kernel buffer, that when rsyslog starts, it fills up the whole capacity of /dev/xconsole with useless debugging early boot info.

And I am mostly interested in the late stage boot messages, and these are missing. Only after I cat /dev/xconsole for the first time, do I free up the space, and new messages can come.

I added this ugly hack in /etc/rsyslog.conf:

:msg, startswith, "\[    "              stop
*.*    |/dev/xconsole

This basically discards all early messages that start with single digit second counter [ 0.000000], but accept all later messages, ie [ 14.348189]

This works, but I consider it dirty workaround.

So, how can I get rid of the early boot messages that I am not interested in, and log the useful init boot script messages?

Answer 1

The syslog process typically picks up messages from three sources:

1. Applications, via /dev/log
2. Network via port 514
3. The kernel, via /proc/kmsg

Now the first... traditionally any application that sends to /dev/log before a syslogd process starts will have their messages lost. systemd confuses things a little because it can listen on that socket instead. And then syslogd can read from systemd. But this doesn't appear to be your concern.

The second is clearly not relevant to your question.

So we focus on the third. These are the lines that begin with [...], where the number is a timestamp from after boot. These are messages created by the kernel, and not user space applications.

Kernel messages are put into a "ring buffer", and when rsyslog starts it will read this buffer to put the data... wherever the config files say put it.

Under normal circumstances this is good. You want a log of all the kernel boot messages, just in case something doesn't start properly.

But if you don't care about them then you can use the command dmesg --clear. That clears the "ring buffer". So if you put that in the boot sequence before you start rsyslog then the daemon will not read any kernel messages; the buffer has been cleared.

Now how you do that is very dependent on the distribution you're running. With traditional sysvinit you might want to create a RC file that runs just before the syslog start script. With systemd you might want to create a unit with a prereq on the syslog process.

Answer 2

This answer is probably very late and OP might not have the need any more but it might serve someone else.

The answer from Stephen Harris is very good but like OP, dmesg --clear does not seem to work for me. However, simply read from /proc/kmsg with something like cat /proc/kmsg > /dev/null before the start of rsyslog does the job.

Note: I would have answered with a comment but I do not have the required reputation.

Answer 3

Clearing the messages from dmesg/kmsg is a bad idea and is likely to bite you in the long run. You might not want to see those messages on xconsole, but they may be important for debugging when stored in other logs.

Probably the best solution here is to clear xconsole early in the boot process, some time immediately after syslog starts but before other "late stage boot messages" are collected.

If some of your "late stage boot messages" are eaten by this process, then you might want to add a parser to the clearing so that it stops either after the last kernel line you want to eat or after the first late stage line you don't want to clear.

You could inject this "xconsole clear" by prioritizing it immediately after rsyslogd or by adding it to the rsyslogd init script if you feel comfortable with that, although changes may get stepped on by updates.