0

Why does journaltcl logs start only today when the last reboot was done 5 days before ? :

$ journalctl -e | grep Logs.begin
-- Logs begin at Tue 2022-09-06 09:42:37 CEST, end at Tue 2022-09-06 11:04:27 CEST. --
$ last reboot | head -1
reboot   system boot  3.10.0-1160.62.1 Thu Sep  1 23:46 - 11:04 (4+11:18)
$ journalctl -u scality-sfused
-- No entries --
$ egrep -v "^$|^#" /etc/systemd/journald.conf
[Journal]
RateLimitInterval=2s
RateLimitBurst=5000
ForwardToSyslog=yes
$ systemctl status scality-sfused | grep Warning
Warning: Journal has been rotated since unit was started. Log output is incomplete or unavailable.
$ ls /etc/logrotate.d/scality-sfused
ls: cannot access /etc/logrotate.d/scality-sfused: No such file or directory
$

EDIT0 : Here is a the /var/lib/logrotate/logrotate.status for today :

$ cat /var/lib/logrotate/logrotate.status | grep $(date +%Y-%-m-%-d)
"/var/log/scality/node/node-10.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-1.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-5.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-5.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-3.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-9.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-9.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-7.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-11.log" 2022-9-6-3:11:1
"/var/log/scality-biziod.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-12.log" 2022-9-6-3:11:1
"/var/log/scality/backup/scality-backup.log.err" 2022-9-6-3:11:1
"/var/log/scality/node/node-11.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-2.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-2.log" 2022-9-6-3:11:1
"/var/log/scality/backup/scality-backup.log.out" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-6.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-6.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-4.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-8.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-12.log" 2022-9-6-3:11:1
"/var/log/scality/node/tier1sync-node-3.log" 2022-9-6-3:11:1
"/var/log/scality-srebuildd.log" 2022-9-6-3:11:1
"/var/log/scality/node/tier1sync-node-7.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-12.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-3.log" 2022-9-6-3:11:1
"/var/log/scality-sagentd.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-3.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-1.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-7.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-7.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-5.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-9.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-10.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-4.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-4.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-2.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-8.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-8.log" 2022-9-6-3:11:1
"/var/log/scality/node/node-6.log" 2022-9-6-3:11:1
"/var/log/scality/node/chord-node-10.log" 2022-9-6-3:11:1
"/var/log/scality/node/chunkapi-node-11.log" 2022-9-6-3:11:1
$

EDIT1 : Here is the /etc/anacrontab file :

$ cat /etc/anacrontab
# /etc/anacrontab: configuration file for anacron

# See anacron(8) and anacrontab(5) for details.

SHELL=/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root
# the maximal random delay added to the base delay of the jobs
RANDOM_DELAY=45
# the jobs will be started during the following hours only
START_HOURS_RANGE=3-22

#period in days   delay in minutes   job-identifier   command
1       5       cron.daily              nice run-parts /etc/cron.daily
7       25      cron.weekly             nice run-parts /etc/cron.weekly
@monthly 45     cron.monthly            nice run-parts /etc/cron.monthly
8
  • Don't you think your traces answer your question : Journal has been rotated since unit was started. ?
    – MC68020
    Sep 6, 2022 at 9:30
  • @MC68020 I just updated my question : ls /etc/logrotate.d/scality-sfused ls: cannot access /etc/logrotate.d/scality-sfused: No such file or directory
    – SebMa
    Sep 6, 2022 at 9:35
  • What does cat /var/lib/logrotate.status tell ?
    – MC68020
    Sep 6, 2022 at 9:44
  • @MC68020 Added it in EDIT0.
    – SebMa
    Sep 6, 2022 at 13:28
  • 2
    ...The reason for the journald content to be "rotated" is that it has limits set regarding the total volume of log data kept on disk (and that is in normally in /var/log/journal, if using persistent storage).
    – goldilocks
    Sep 6, 2022 at 14:55

1 Answer 1

1

A glance at the logrotate status output does seem to confirm that the scality app's logs were rotated. However, there is a bit of confusion here because there are two sets of logs to consider, and what logrotate does does not affect what's returned by journalctl, nor are journald's own files managed by logrotate.

Journald Logs

man systemd-journald.service lists 5 sources for the logs kept and reported by journald:

  1. Kernel log messages

    These are not output from applications, although they may be influenced by the behaviour of such -- as per the name, they are messages from the kernel. It's the same stuff you can see with dmesg.

  2. Simple system log messages, via the libc syslog(3) call

    This is a traditional logging method that applications including persistent services can make use of. If you are running a traditional syslog implementation instead of or as well as systemd-journald (they can work nicely together), these are output to a set of files in /var/log such as messages and syslog. If not, they are still caught by journald and show up in its service hierarchy. Note that this does not allow for application specific files unless the logger itself is set up to do that, which is beyond the control of the application. Stuff that's sent to syslog is usually tagged with the application name, but it is all lumped together -- which can be very useful, since kernel messages are usually also captured by syslog, so syslog is an interleaved record of system events in real time.

    However, many applications do not use this at all, and as far as I am aware there is no strong convention for or against that. Note that systemd itself logs to syslog via journald, syslog being something journald can feed.

  3. Structured system log messages via the native Journal API

    This is presumably journald's version of the syslog command, meaning it would have to be explicitly used in application source code. How widespread this is I don't know, but probably not much beyond core linux centric things such as desktop environments, package managers, etc.

  4. Standard output and standard error of service units.

    Capturing these is a default behaviour of systemd-journald, but they can also be redirected via the service file to various places.

    In my experience, most complex service applications do not use this for much.

  5. Audit records, originating from the kernel audit subsystem

    In context these are really a subcategory of #1.

  6. This one isn't explicitly in the manpage, but unit specific journalctl output does obviously include messages from systemd that pertain to the unit in question (eg. Starting XXXXX...).

None of these include that bunch of stuff in /var/log/scality, meaning those files have nothing to do with journald or syslog.

Application Logs

I'm not a scality user, but certainly the stuff in /var/log/scality will be stuff that is created directly by the application, and not anything that's been captured externally.

An obvious reason for these things not being used by systemd or journald is that programs write data to all kinds of files, and without some configuration protocol to guide the process, external entities like journald have no way of knowing what is logging data and what is not. There's not much justification for such a protocol either since, as per above, there are already mechanisms in place for applications to make use of syslog etc.

Things that write a lot of their own log files usually (and this may be the work of distro packagers) arrange for logrotate to keep them trim. Logrotate predates systemd and has nothing to do with journald. Journald maintains its own persistent storage (if any).

In short, the orignal message about how the "Journal has been rotated since unit was started" is not because of logrotate. This is part of journald's own operation. Further, the scality logs in /var/log/scality are not managed by journald, but by scality itself.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .