0

I have a systemd service unit for sshd:

$ systemctl status sshd
* ssh.service - OpenBSD Secure Shell server
   Loaded: loaded (/lib/systemd/system/ssh.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2020-09-30 18:54:10 UTC; 6s ago
  Process: 13923 ExecReload=/bin/kill -HUP $MAINPID (code=exited, status=0/SUCCESS)
  Process: 13918 ExecReload=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
  Process: 6287 ExecStartPre=/usr/sbin/sshd -t (code=exited, status=0/SUCCESS)
 Main PID: 6296 (sshd)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/ssh.service
           `-6296 /usr/sbin/sshd -D

Sep 30 18:54:10 machine systemd[1]: Starting OpenBSD Secure Shell server...
Sep 30 18:54:10 machine sshd[6296]: Server listening on 0.0.0.0 port 22.
Sep 30 18:54:10 machine sshd[6296]: Server listening on :: port 22.
Sep 30 18:54:10 machine systemd[1]: Started OpenBSD Secure Shell server.

I have some *.journal files that it wrote:

$ find /var/log/journal -type f -name 'system@*.journal' -exec grep -l 'OpenBSD' {} +
/var/log/journal/0123456789abcdef0123456789abcdef/system@0123456789abcdef0123456789abcdef-000000000100bfdd-0005ac0c092a655f.journal
/var/log/journal/0123456789abcdef0123456789abcdef/system@0123456789abcdef0123456789abcdef-000000000102d6c3-0005ac2251abe7d9.journal
/var/log/journal/0123456789abcdef0123456789abcdef/system@0123456789abcdef0123456789abcdef-00000000010fc6d5-0005ac9be551a106.journal
/var/log/journal/0123456789abcdef0123456789abcdef/system@0123456789abcdef0123456789abcdef-0000000000e043b9-0005aad981c2d167.journal
/var/log/journal/0123456789abcdef0123456789abcdef/system@0123456789abcdef0123456789abcdef-00000000010d9bf1-0005ac875e1241e7.journal

Given the former, how do I find the latter?

5

Journal files are not sorted by unit. The unit issuing the journal entry is simply metadata associated with the entry. Journal files are also not always in /var. They could also be found in /run if configured as non-persistent.

The correct way to extract journal entries for a specific unit is to use journalctl -u <unit_name> or to use the sd-journal API. In addition, the journal is a binary file so it's not exactly in a format that is parse-friendly. I'm not even sure that the internal format of the journal follows a stable specification. journalctl or the API is definitely the way to read these files.

If you are asking this question to back-up (or send for troubleshooting) a specific file, you'll need to treat all of these files as a single unit.


Since you asked specifically about the file-naming convention of these journal files, I spent some time investigating this. After a fresh boot, the only file in /run/log/journal was /run/log/journal/c13e4a2f54334a95891a6a471db3b7e0/system@edd0c14e44a240038601194807d5c28e-0000000000000001-0005b08ccb2c37b3.journal.

I broke that down into the following to try and understand the naming convention:

/run/                                  <-- run/var
  log/journal/                         <-- standard location 
    c13e4a2f54334a95891a6a471db3b7e0/  <-- _MACHINE_ID
    system@                            <-- system/user
    edd0c14e44a240038601194807d5c28e-  <-- Some bus/boot/session ID?
    0000000000000001-                  <-- Sequence number (also looks like offsets as 4-KB blocks in hex)
    0005b08ccb2c37b3                   <-- Timestamp (time since epoch in hex)
    .journal                           <-- Extension

When I look at a machine which has been running for a while, all system@*.journal files are 80M while all user@*.journal files are 8.0M. With such consistent sizing, it really suggests that the file splits are simply log rotations. I can also confirm that by looking at the modification dates (as we go forward in sequence, I see the dates increasing).


To figure some of these things out (like _MACHINE_ID) I had to look into the raw meta-data associated with a journal entry. There is more meta-data than what you can get with journalctl. I wrote a quick program with the sd-journal API and printed "Hello" from a service. I found it in the journal with the following data. It's amazing how much meta-data is associated with each printf.

_UID=0
_GID=0
_CAP_EFFECTIVE=3fffffffff
_TRANSPORT=stdout
_STREAM_ID=2ee82c395817429b975770acb1306d11
SYSLOG_IDENTIFIER=counter
_PID=6523
_COMM=counter
_EXE=/usr/local/bin/counter
_CMDLINE=/usr/local/bin/counter
_SYSTEMD_CGROUP=/system.slice/counter.service
_SYSTEMD_UNIT=counter.service
_SYSTEMD_INVOCATION_ID=711962f5d7bf487b9a262b824c297a42
MESSAGE=Hello
PRIORITY=6
SYSLOG_FACILITY=3
_BOOT_ID=bf0d96cbab2144e3a8544b0e8a0eacc6
_MACHINE_ID=c13e4a2f54334a95891a6a471db3b7e0
_HOSTNAME=simswe24
_TRANSPORT=journal
_SELINUX_CONTEXT=unconfined

If you're interested in the sd-journal program I wrote to investigate this, it's here:

// Compile with `gcc journalctl.c -lsystemd`
#include <systemd/sd-journal.h>
#include <stdio.h>
int main()
{
    // Open the log
    sd_journal* journal;

    if ( sd_journal_open(&journal, SD_JOURNAL_LOCAL_ONLY) < 0 )
    {
        return 1; // Couldn't open journal
    }

    // Subscribe to specific logs:
    // Messages from the service itself
    int r = sd_journal_add_match(journal, "_SYSTEMD_UNIT=counter.service", 0);
    r = r ? r : sd_journal_add_disjunction(journal);
    
    // Messages from PID 1 (systemd) about this service
    r = r ? r : sd_journal_add_match(journal, "_PID=1", 0);
    r = r ? r : sd_journal_add_match(journal, "UNIT=counter.service", 0);
    r = r ? r : sd_journal_add_disjunction(journal);

    if ( r < 0 )
    {
        return 1; // Could not subscribe to all journal entries
    }

    r = sd_journal_seek_head(journal);
    if ( r < 0 )
    {
        return 1; // Problem finding the head of the log
    }


    // Read the log
    int skip_seek=1;
    while( skip_seek || (sd_journal_next(journal) > 0 ) ) 
    {
        const void* data = 0;
        size_t length;
        skip_seek = 0;

        sd_journal_restart_data(journal);

        SD_JOURNAL_FOREACH_DATA(journal, data, length)
        {
            printf("%s\n", data);
        };
    }

    sd_journal_close(journal);

    return 0;
}

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.