41

I spent a few days writing a python script, and creating a systemd unit file for it. During testing, the script logged a lot of errors to journald. I would like to clear those errors from journald now that I'm done.

There are several ways to clear the entire journal, as described here: How to clear journalctl including using journalctl --vacuum-time=2d, using journalctl --vacuum-size=500M, and temporarily setting SystemMaxUse= in /etc/systemd/journald.conf to a very low value.

All of these appear to clear the entire journal, effecting all units. I just need to clear the entries for a single unit. Is this possible?

1

4 Answers 4

18

Use my Python 3 program copy_journal.py on the journal files in /var/log/journal from which you want to remove entries.

For instance, to make a copy of system.journal without log entries for NetworkManager.service:

$ journalctl --file=system.journal | wc
    167    1934   18825
$ journalctl --file=system.journal | grep -v NetworkManager | wc
     77     881    8421
$ python3 copy_journal.py --remove-unit=NetworkManager.service system.journal system-without-nm.journal
$ journalctl --file=system-without-nm.journal | wc
     77     881    8421
1
  • Wow, huge work on this package!
    – Joël
    Commented May 31, 2020 at 8:45
14

I was looking for a way to remove a specific message from the systemd journal. This didn't seem like an uncommon scenario (e.g. accidentally running a wrong command or having lots of spam).

It appears to be infeasible to redact specific entries by modifying an existing file, because the systemd Journal format is primarily designed to support appending, which involves optional compression, hashing, updating counters, indexing references to offsets and hashes. I tried this at first, but would not recommend anyone to do this.

To remove specific entries, I found that the next best thing is to generate a new log file based on the existing log file (e.g. the Python script from the other answer does this). Systemd actually ships the necessary tools to easily modify log entries using this principle.

Short version

In essence, export the journal to a text format, and then convert it back to a journal:

  • journalctl --file=system.journal -o export | /lib/systemd/systemd-journal-remote --output=result.journal

This mechanism can easily be extended by piping the initial output to a program/script (such as awk, sed, etc.) to do automatic filtering before piping it to systemd-journal-remote.

A more manual approach is to write the text to a file, modify the file by hand with a text editor and then converting it back to a .journal file. This is explained in more detail below.

Long version

Here is the longer version, a step-by-step guide to modify an existing journal (tested on ArchLinux, using an Ubuntu live USB, systemd 246).

  1. First, ensure that the journal has been closed. For example by shutting down your computer.

  2. Boot a different device (via a bootable live USB for example). This is to be able to make changes without inadvertently impacting the journal.

  3. To minimize accidental side effects, mount the original disk as read-only. For example:

    $ sudo mount -o ro /dev/sda1 /mnt
    

    another example, if you're using LUKS to encrypt the device:

    $ sudo cryptsetup luksOpen /dev/sda1 whateverName
    $ sudo mount -o ro /dev/mapper/whateverName /mnt
    
  4. If your current device doesn't have systemd / journalctl, use a chroot to be able to use the binaries from your read-only disk. The minimum to get this to work is as follows:

    $ sudo mount --bind /tmp /mnt/tmp
    $ sudo mount --bind /tmp /mnt/var/tmp
    $ sudo mount -t proc /proc /mnt/proc
    $ sudo chroot /mnt
    


  1. When you are ready, export the journal.

    $ journalctl --file=system.journal -o export > /tmp/export.txt
    
  2. Edit the export to redact, add, modify or remove entries. You can do this outside the chroot if you want to, e.g. with vi -b /tmp/export.txt.
    Example of the output: https://www.freedesktop.org/wiki/Software/systemd/export/

  3. When you have made the desired modifications, convert it back to a journal using systemd-journal-remote. This binary is not in bin / $PATH, so you need to use the absolute path to the program's location:

    $ /lib/systemd/systemd-journal-remote --output=/tmp/result.journal /tmp/export.txt
    
  4. Verify that the journal has the desired output, e.g. by paging through the journal with:

    $ journalctl --file=/tmp/result.journal
    

    If you are satisfied, proceed with the next step. Otherwise repeat from step 6.



  1. Exit the chroot (CtrlD or simply exit) and undo step 4, if needed:

    $ sudo umount /mnt/proc
    $ sudo umount /mnt/tmp
    $ sudo umount /mnt/var/tmp
    
  2. Now we are about to overwrite the original journal.
    If you have mounted the volume as read-only (step 3), remount as writable.

    $ sudo mount -o remount,rw /mnt
    

    To avoid unwanted changes in file attributes or ownership, just replace the content instead of copying the file.

    # cat /tmp/result.journal > /mnt/var/log/journal/MACHINE_ID_HERE/system.journal
    

    or if you aren't running in a root shell, use sudo tee outfile >/dev/null, like this:

    $ </tmp/result.journal sudo tee /mnt/var/log/journal/MACHINE_ID_HERE/system.journal >/dev/null
    
  3. Unmount the disk (from step 3 / step 10) if needed.

    $ sudo umount /mnt
    

    If you had opened an encrypted volume (at step 3), lock it again:

    $ sudo cryptsetup luksClose whateverName
    
2
  • 2
    I would say this is unnecessarily complicated – instead of having to use a liveCD in order to work on the "live" system.journal file, it'd be much easier to trigger a log rotation via journalctl and then do anything you like to the no-longer-active system@[id].journal. Commented Oct 25, 2022 at 15:23
  • @u1686_grawity Indeed, this (otherwise great) answer can be made much safer and simpler by replacing steps 1-4 with journalctl --system --rotate, replacing system.journal in steps 5 with the rotated name and replacing steps 9-11 with just a simple copy/move (like what's now in step 10, but to the rotated name)
    – TooTea
    Commented Oct 25, 2023 at 8:29
2

I copied here my answer from duplicated question on stackoverflow

I couldn't find the specific version it became possible, journaltcl documentation doesn't even mention that --vacuum-time (or -size or -files) can be combined with --unit but I have ver:249, the vacuuming came in ver:218, and unit filter came in ver:195.

TLDR: Old and roundabout methods are no longer necessary:
To clean up all logs for a unit:
journalctl --vacuum-time=1s --unit=your.service

If you don't mind rotating logs of everything else AND you still find logs from your unit (meaning its last log haven't rotated yet) you can rotate in one go:
journalctl --rotate --vacuum-time=1s --unit=your.service

1
  • This worked for me, I had a known issue with bluetooth "Two services allocated for the same bus name org.bluez", that required checking the log with journalctl | grep -i bluetooth but it took over a minute to scroll to the end each time. using this answer with --unit=bluetooth.service cut the log down. Commented May 22 at 21:28
0

I do not think think its possibble, because according to journalctl man page "Output is interleaved from all accessible journal files, whether they are rotated or currently being written".

You could however filter it by a unit - 'journalctl -u some.service' and then try to remove the log entries from the journal with your own script using cron.

1
  • 10
    It sounds like you are implying that there is a way to delete individual log entries from journald. Is there one? That would be very helpful.
    – Tal
    Commented Mar 29, 2016 at 16:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .