rsyslog not logging kern messages

Question

I'm using Ubuntu 20.04 with rsyslog and up-to-date packages. I want to log messages from iptables.

I can see those messages with dmesg - it shows the data I expect.

deploy$ dmesg -x
kern  :warn  : [37464969.767649] iptables: Incoming netperf IN=venet0 OUT= MAC= SRC=x.x.x.x DST=y.y.y.y LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=43720 DF PROTO=TCP SPT=46221 DPT=12865 WINDOW=29200 RES=0x00 SYN URGP=0
kern  :warn  : [37464969.767679] iptables: Incoming netperf IN=venet0 OUT= MAC= SRC=x.x.x.x DST=y.y.y.y LEN=60 TOS=0x00 PREC=0x00 TTL=51 ID=34689 DF PROTO=TCP SPT=58437 DPT=12865 WINDOW=29200 RES=0x00 SYN URGP=0

BUT... those messages are never saved in any log file in /var/log directory.

As far as I can tell, my /etc/rsyslog.conf (below) file is unchanged from the original, and the default rules (in /etc/rsyslog.d/50-default.conf - further below) should cause the kern messages to be written to /var/log/kern.log. The only other file in /etc/rsyslog.d/ is postfix.conf (next).

File permissions don't seem to be surprising... (Bottom two reports)

Thoughts? What am I missing? Thanks.

---

/etc/rsyslog.conf

# /etc/rsyslog.conf configuration file for rsyslog
#
# For more information install rsyslog-doc and see
# /usr/share/doc/rsyslog-doc/html/configuration/index.html
#
# Default logging rules can be found in /etc/rsyslog.d/50-default.conf


#################
#### MODULES ####
#################

module(load="imuxsock") # provides support for local system logging
#module(load="immark")  # provides --MARK-- message capability

# provides UDP syslog reception
#module(load="imudp")
#input(type="imudp" port="514")

# provides TCP syslog reception
#module(load="imtcp")
#input(type="imtcp" port="514")

# provides kernel logging support and enable non-kernel klog messages
#module(load="imklog" permitnonkernelfacility="on")

###########################
#### GLOBAL DIRECTIVES ####
###########################

#
# Use traditional timestamp format.
# To enable high precision timestamps, comment out the following line.
#
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

# Filter duplicated messages
$RepeatedMsgReduction on

#
# Set the default permissions for all log files.
#
$FileOwner syslog
$FileGroup adm
$FileCreateMode 0640
$DirCreateMode 0755
$Umask 0022
$PrivDropToUser syslog
$PrivDropToGroup syslog

#
# Where to place spool and state files
#
$WorkDirectory /var/spool/rsyslog

#
# Include all config files in /etc/rsyslog.d/
#
$IncludeConfig /etc/rsyslog.d/*.conf

/etc/rsyslog.d/50-default.conf

#  Default rules for rsyslog.
#
#           For more information see rsyslog.conf(5) and /etc/rsyslog.conf

#
# First some standard log files.  Log by facility.
#
auth,authpriv.*      -/var/log/auth.log
*.*;auth,authpriv.none      -/var/log/syslog
#cron.*          -/var/log/cron.log
#daemon.*           -/var/log/daemon.log
kern.*  -/var/log/kern.log
#lpr.*              -/var/log/lpr.log
mail.*              -/var/log/mail.log
#user.*             -/var/log/user.log

#
# Logging for the mail system.  Split it up so that
# it is easy to write scripts to parse these files.
#
#mail.info          -/var/log/mail.info
#mail.warn          -/var/log/mail.warn
mail.err         -/var/log/mail.err

#
# Some "catch-all" log files.
#
#*.=debug;\
#   auth,authpriv.none;\
#   news.none;mail.none -/var/log/debug
#*.=info;*.=notice;*.=warn;\
#   auth,authpriv.none;\
#   cron,daemon.none;\
#   mail,news.none      -/var/log/messages

#
# Emergencies are sent to everybody logged in.
#
*.emerg             :omusrmsg:*

#
# I like to have messages displayed on the console, but only on a virtual
# console I usually leave idle.
#
#daemon,mail.*;\
#   news.=crit;news.=err;news.=notice;\
#   *.=debug;*.=info;\
#   *.=notice;*.=warn   /dev/tty8

/etc/rsyslog.d/postfix.conf

# Create an additional socket in postfix's chroot in order not to break
# mail logging when rsyslog is restarted.  If the directory is missing,
# rsyslog will silently skip creating the socket.
$AddUnixListenSocket /var/spool/postfix/dev/log

ls -al /etc/rsys*

deploy:/etc$ ls -al /etc/rsys*
-rw-r--r-- 1 root root 1383 Oct 12 10:43 rsyslog.conf

rsyslog.d:
total 24
drwxr-xr-x   2 root root    4096 Feb 16 08:28 .
drwxr-xr-x 114 root deploy 12288 Feb 15 18:41 ..
-rw-r--r--   1 root root    1124 Feb 16 07:48 50-default.conf
-rw-r--r--   1 root root     242 Jun 19  2020 postfix.conf

ls -al /var/log/kern.log

deploy:/etc$ ls -al /var/log/kern.log
-rw-r----- 1 syslog adm 0 Feb 15 18:50 /var/log/kern.log

sudo systemctl status systemd-journald.service

deploy:~$ sudo systemctl status systemd-journald.service
● systemd-journald.service - Journal Service
     Loaded: loaded (/lib/systemd/system/systemd-journald.service; static; vendor preset: enabled)
     Active: active (running) since Tue 2021-02-16 09:02:16 EST; 45min ago
TriggeredBy: ● systemd-journald.socket
             ● systemd-journald-audit.socket
             ● systemd-journald-dev-log.socket
       Docs: man:systemd-journald.service(8)
             man:journald.conf(5)
   Main PID: 179 (systemd-journal)
     Status: "Processing requests..."
      Tasks: 1 (limit: 38354)
     Memory: 52.7M
     CGroup: /system.slice/systemd-journald.service
             └─179 /lib/systemd/systemd-journald

Jan 31 21:42:14 example.com systemd-journald[35283]: Journal stopped
Jan 31 21:42:21 example.com systemd-journald[180]: Journal started
Jan 31 21:42:21 example.com systemd-journald[180]: Runtime Journal (/run/log/journal/77578ccc822c4f1aaa10385353e145a7) is 2.5M, max 20.4M, 17.9M free.
Jan 31 21:42:22 example.com systemd-journald[180]: Time spent on flushing to /var/log/journal/77578ccc822c4f1aaa10385353e145a7 is 29.543ms for 19 entries.
Jan 31 21:42:22 example.com systemd-journald[180]: System Journal (/var/log/journal/77578ccc822c4f1aaa10385353e145a7) is 32.0M, max 4.0G, 3.9G free.
Feb 16 09:02:10 example.com systemd-journald[180]: Journal stopped
Feb 16 09:02:16 example.com systemd-journald[179]: Journal started
Feb 16 09:02:16 example.com systemd-journald[179]: Runtime Journal (/run/log/journal/77578ccc822c4f1aaa10385353e145a7) is 2.5M, max 20.4M, 17.9M free.
Feb 16 09:02:16 example.com systemd-journald[179]: Time spent on flushing to /var/log/journal/77578ccc822c4f1aaa10385353e145a7 is 54.224ms for 20 entries.
Feb 16 09:02:16 example.com systemd-journald[179]: System Journal (/var/log/journal/77578ccc822c4f1aaa10385353e145a7) is 160.0M, max 4.0G, 3.8G free.
deploy:~$

Answer 1

Update: It's working now. While troubleshooting this, I made a huge number of changes, but I believe the problem was that the line below was commented out in /etc/rsyslog.conf Uncommenting it causes the log entries to appear as expected.

#module(load="imklog" permitnonkernelfacility="on")

In the process I collected a bunch of facts about rsyslog that I put here for the next poor person (including me) who has this kind of trouble:

1. iptables always sends its log messages through a kern.* facility
2. rsyslog config is in /etc/rsyslog.conf; it frequently points to separate files in /etc/rsyslog.d that provide additional rules for various kinds of log files.
3. (As I learned), loading the kernel logging module with module(load="imklog" ...) is required. Other facilities may need other modules.
4. /etc/rsyslog.d/50-default.conf has a line for directing kern.* messages to a particular log file.
5. On Ubuntu 20.04 (at least) you need to use sudo service rsyslog restart after config changes
6. rsyslogd -N1 checks for errors in the rsyslog configuration
7. journald seems unrelated to any of the problems I had.

Things I could NOT verify: It doesn't mean it doesn't work as documented, just that I probably did it wrong.

1. I couldn't make a file /etc/rsyslog.d/10-iptables.conf override the 50-default.conf file in the directory (to direct certain messages to a separate file).