I have a systemd service that I install via debian package using the apt
command on a raspberry pi 4. I've been using journalctl -xef
to keep track of all log output for debugging purposes while installing the package and running the service.
I'm not sure how I managed this, but at some point today, apt
's log output as well as the output of my package's maintainer scripts stopped showing up in the journal. I did have several typos in one of my package's maintainer scripts and I have a tendency to fat finger the keyboard or press the hotkeys sometimes, but I can't find anything in particular that might've caused the issue. It seems that the journal defaults to in-memory storage as well, which is a bummer, because the first thing I tried to fix the issue was a reboot. So now I guess I can't even look back at old journal entries to see if there was something in there that might indicate what had happened.
I also thought perhaps some of the changes I'd made today in the service or package were causing the problem, so I tried installing an older proof-of-concept service/package and that too does not write correctly to the journal.
I've tried to find a way to just reinstall or repair the apt/apt-get/dpkg
installation, but haven't been very successful. Most questions are related to repairing a package installation using apt
, not repairing the installation of apt
itself. I also tried to see if there was a way to configure where apt
writes its logs, but most questions along those lines are related to redirecting stdout/stderr
when running an apt
command.
I'm very new to linux and all of the technologies listed above, so I'm not sure how to debug/diagnose this further.
Is there a way to repair/restore the apt/apt-get/dpkg
installations or configurations on a raspberry pi or is my only option to try to restore to factory defaults or re-install the entire OS?
Update
Today I learned of the /var/log
directory. I haven't found any indication within those logs of what caused this issue, but it does appear that the the problem also extends to syslog
, messages
, user.log
and daemon.log
(i.e. output from apt/apt-get
and from package maintainer scripts is missing from these as well). I can see several apt install/upgrade/purge
operations successfully logged in those files and then no more. I can tell there should be more entries logged, however, because my package calls systemctl daemon-reload
and I can see systemd[1]: Reloading
in the logs. From browsing /var/log/dpkg.log
, it seems that none of those entries were every shown in journald
, so that seems to still be fine. Logs from /var/log/apt/term.log
are definitely missing from journald
, but logs from /var/log/auth.log
are still present.