How can I use journalctl to time-filter and view non-journal files?

Question

I'm trying to search through dozens of logfiles that were not sent to systemd-journald, and would like to filter the results by time, the way journalctl's -S (since) and -U (until) do.

• Option 1: write a program that reads lines, parses timestamps, prints filtered output
• Option 2: use /usr/bin/dateutils.dgrep from the Ubuntu dateutils package
• Option 3: convert the logfiles into journal format and use journalctl --root $dir -S xxx -U yyy

I started on #1, realized I was reinventing the wheel, tried #2, found that it was incredibly slow, and then looked at #3, mainly because it's fast, polished, and has had many "eyeballs" (especially "paid eyeballs") looking at it.

So, is there a tool that will convert syslog-style logfiles into journal files such that the entries can be filtered by time? Systemd-cat won't work, because it's not allowed to forge timestamps: the resulting journal file will say everything happened at roughly the same time.

Any suggestions?

Thanks!

Edit:

The log files are > 100MB, there are close to a hundred of them, the timestamps are plain syslog-style "MMM DD HH:MM:SS", and I just want to say "show me the log entries between these two times".

Option #1 was "write a program to do it", but I'm certain this wheel has been reinvented many times already; I'd feel foolish adding one more (mediocre) wheel to the pile.

Option #2 hadn't finished even one file before I gave up after half an hour. Essentially, I want a fast general solution that can be reused, and journalctl looked like a good part of the answer.

I reckon that journalctl takes advantage of the monotonically increasing timestamps of log files and uses binary search to find records in a given time range. The dateutils.dgrep (dategrep) program isn't allowed to make that assumption, which makes it slower than molasses in January.

Edit2:

It looks like writing a short script to parse the input logs, emit the records in journal-export format, and use systemd-journal-remote to generate a journal file would work:

journalctl -n 10 -o export > journal.txt
/lib/systemd/systemd-journal-remote -o test.journal journal.txt
journalctl --file=test.journal -S "yyyy-mm-dd hh:mm:ss" -U "yyyy-mm-dd hh:mm:ss"
   (log entries for time range shown)
journalctl --verify --file=test.journal

PASS: test.journal 

Initially I thought that the __CURSOR lines might contain sealing checksums, but deleting them had no obvious effect. It appears that Forward Secure Sealing is not enabled by default on Ubuntu 20.04, but journalctl does detect if log entries are not monotonically increasing:

rm -f test.journal
/lib/systemd/systemd-journal-remote -o test.journal journal.txt
/lib/systemd/systemd-journal-remote -o test.journal journal.txt
journalctl --verify --file=test.journal

3947a0: Entry timestamp out of synchronization                                                                                           
File corruption detected at test.journal:3947a0 (of 8388608 bytes, 44%).                                                                 
FAIL: test.journal (Bad message)

It seems it will be fairly easy to import any arbitrary logfile into journal format and access it with journalctl.

Answer 1

Option 1 - filter log files by time (skip journal conversion)

I'm trying to search through dozens of logfiles that were not sent to systemd-journald, and would like to filter the results by time, the way journalctl's -S (since) and -U (until) do.

If you only want to view the syslog-style log files in a particular datetime range, then try super-speedy-syslog-searcher. This uses features of super-speedy-syslog-searcher and skips the use of the systemd journal log service.

Assuming you have rust installed, run

$ cargo install super_speedy_syslog_searcher

Then run s4 on the directory containing the log files

$ s4 /logs

To confine to particular datetime range, pass -a and -b options to filter by datetime. An example processing of log files using datetime filtering

$ cat /tmp/logs/kernel.log
<6>Jan  1 13:58:25 HOST kernel: [44033.150723] eth0: link up (100Mbps/Full duplex)
<6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888

$ cat /tmp/logs/syslog.1
<29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up
<86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received

$ s4 -a "2023-01-02T00:00:00Z" -b "2023-01-03T00:00:00Z" /tmp/logs/
<6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888

---

Option 2 - convert log files into journal files

is there a tool that will convert syslog-style logfiles into journal files such that the entries can be filtered by time?

Overview

A sequence of operations to convert the syslogs would be:

1. use super-speedy-syslog-searcher to reprint logs with specially formatted datetime.
2. use journalctl to convert each syslog message to a basic logger message in line-oriented journal export format
3. use systemd-journal-remote to import the messages

1. use super-speedy-syslog-searcher to reprint a log file

Given a log file /tmp/logs/syslog.1 with syslog-style messages

<29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up
<86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received

Using super-speedy-syslog-searcher (s4), reprint lines with file name (-n) and leading datetime as a Unix Epoch in microseconds (-d '%s000000') in UTC timezone (-u).

$ s4 -n -u -d '%s000000' /tmp/logs/syslog.1
syslog.1:1672610305000000:<29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up
syslog.1:1672786836000000:<86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received

2. use journalctl to convert a syslog message to a journal format message

Here is an example of creating a minimal contrived journal format log message using logger:

$ echo "SYSLOG_IDENTIFIER=logger
MESSAGE=one journal logger message
MESSAGE_ID=$(uuidgen)" | logger --journald -s

$ journalctl -n1 -t logger
Feb 03 00:48:01 HOST logger[445736]: one journal logger message

Here is that message exported to journal export format

$ journalctl -n1 --identifier=logger --output=export
__CURSOR=s=4a35367b4dd3403b948f1a5ff2fb3515;i=474833;b=5285cccfe2f9481ca12589f5511b8c57;m=dd9ee>
__REALTIME_TIMESTAMP=1675414081923660
__MONOTONIC_TIMESTAMP=951851827460
_BOOT_ID=5286cccfe2f9481ca1ea8cf5221b8c5e
_UID=0
_GID=0
_MACHINE_ID=333fa82c526bba518cdc8c2262e6d480
_HOSTNAME=HOST
_TRANSPORT=journal
SYSLOG_IDENTIFIER=logger
MESSAGE=one journal logger message
MESSAGE_ID=4ebd2168-30bf-4258-bf0f-54b5811674e1
_PID=445736
_COMM=logger
_SOURCE_REALTIME_TIMESTAMP=1675414081923606

3. use systemd-journal-remote to import a journal log message

To import a contrived message, resend similar data to the journal service using systemd-journal-remote.

For the log messages, each new journal entry only needs parameters MESSAGE_ID, MESSAGE, _TRANSPORT, __REALTIME_TIMESTAMP, _SOURCE_REALTIME_TIMESTAMP. Set _TRANSPORT to syslog. Override values __REALTIME_TIMESTAMP and _SOURCE_REALTIME_TIMESTAMP with the datetime stamp as microseconds since Unix Epoch. For Jan 1 15:00:36 2023 PST, the value is 1672614036000000.

$ echo "\
__REALTIME_TIMESTAMP=1672614036000000
_TRANSPORT=syslog
MESSAGE_ID=4ebd2168-30bf-4258-bf0f-54b5811674e1
MESSAGE=one journal logger message
SYSLOG_IDENTIFIER=logger
_SOURCE_REALTIME_TIMESTAMP=1672614036000000
" | /usr/lib/systemd/systemd-journal-remote --output=/tmp/tmp.journal -

$ journalctl --output=short-full --file=/tmp/tmp.journal
Sun 2023-01-01 15:00:36 PST logger: one journal logger message

putting 1., 2., and 3. together

Put all this together into a short shell script that will process a directory of log files /tmp/logs into journal files at /tmp/logsj.

Given syslog-style log files:

$ cat /tmp/logs/kernel.log
<6>Jan  1 13:58:25 HOST kernel: [44033.150723] eth0: link up (100Mbps/Full duplex)
<6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888

$ cat /tmp/logs/syslog.1
<29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up
<86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received

Reprinting file kernel.log using s4 looks like:

$ s4 --color=never -n -u -d '%s000000' /tmp/logs/kernel.log | tail -n1
kernel.log:1672689660000000:<6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888

Notice the prepended fields are separated by : (the default).

Process each syslog-style log file under /tmp/logs into a corresponding journal file under path /tmp/logsj. This sets SYSLOG_IDENTIFIER to the file name, excluding file extensions. The created journal file is the name of the source file plus .journal.

#!/usr/bin/env bash

mkdir /tmp/logsj

while read line; do
  echo "line is '${line}'"
  name=$(echo -n "${line}" | cut -f1 -d:)
  ts=$(echo -n "${line}" | cut -f2 -d:)
  mesg=$(echo -n "${line}" | cut -f3- -d:)
  jmesg="\
__REALTIME_TIMESTAMP=${ts}
_TRANSPORT=syslog
MESSAGE_ID=$(uuidgen)
MESSAGE=${mesg}
SYSLOG_TIMESTAMP=${ts}
SYSLOG_IDENTIFIER=${name%%.*}
_SOURCE_REALTIME_TIMESTAMP=${ts}
"
  echo "${jmesg}" | /usr/lib/systemd/systemd-journal-remote "--output=/tmp/logsj/${name}.journal" -
done <<< $(s4 --color=never -n -u -d '%s000000' /tmp/logs/)

Review the new journal files:

$ ls -l /tmp/logsj/
-rw-r----- 1 root root 8388608 Feb  3 14:36 kernel.log.journal
-rw-r----- 1 root root 8388608 Feb  3 14:36 syslog.1.journal

$ PAGER= journalctl --utc --output=short-full --directory=/tmp/logsj/
Sun 2023-01-01 21:58:25 UTC kernel: <6>Jan  1 13:58:25 HOST kernel: [44033.150723] eth0: link up (100Mbps/Full duplex)
Sun 2023-01-01 21:58:25 UTC syslog: <29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up
Mon 2023-01-02 20:01:00 UTC kernel: <6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888
Tue 2023-01-03 23:00:36 UTC syslog: <86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received

Run journalctl on a datetime slice

$ PAGER= journalctl --utc --output=short-full --until "2023-01-01 23:00:00 UTC" --directory=/tmp
/logsj/
Sun 2023-01-01 21:58:25 UTC kernel: <6>Jan  1 13:58:25 HOST kernel: [44033.150723] eth0: link up (100Mbps/Full duplex)
Sun 2023-01-01 21:58:25 UTC syslog: <29>Jan  1 13:58:25 HOST netifd: Network device 'eth0' link is up

$ PAGER= journalctl --utc --output=short-full --since "2023-01-01 23:00:00 UTC" --directory=/tmp
/logsj/
Mon 2023-01-02 20:01:00 UTC kernel: <6>Jan  2 12:01:00 HOST kernel: [194033.150723] loop7: detected capacity change from 0 to 113888
Tue 2023-01-03 23:00:36 UTC syslog: <86>Jan  3 15:00:36 HOST dropbear[23732]: Exit (root): Disconnect received