Problem:
systemd user units journalctl output display "No journal files were found." This only happens for the one sss user I am testing with, all local users seem to work fine. EDIT: sss does not seem to be the issue. Just this user is the problem. Exact command and output:
$journalctl --user -u myservice
No journal files were found.
system env
- RHEL 8.4
- systemd 239 (239-45.el8_4.3)
- kernel: 4.18.0-305.19.1.el8_4.x86_64
Troubleshooting:
I already read these posts and tried troubleshooting found within each:
- https://stackoverflow.com/questions/30783134/systemd-user-journals-not-being-created
- https://bbs.archlinux.org/viewtopic.php?id=193518
- Why does journalctl say "-- no entries --"?
- journalctl --user giving "No journal files were found."
I have tried the following:
- Rebooting.
- Redeploying the host OS.
- Tried creating a local user to test with.
- Made sure to login to all newly created local user account I am working with directly with SSH or terminal initially before continuing testing. Also tested sshing to the sss user account in question which I am able to.
- Restarting journalctl services (
systmectl restart systemd-journald.service) - Made sure /etc/systemd/journald.conf
Storage=persistent. But also tried auto, and volatile, no change with any. - added
export XDG_RUNTIME_DIR=/run/user/$(id -u)in ~/.bashrc for the local user and the user in question. Then sourced the file. - Confirmed linger is enabled.
- Noticed that this problem is only happening one specific user on the box. Created new users, and this problem does not happen.
- User account in question has a different uid from the gid/groups. However I tested adjusting a newly created account with a different uid from its gid/groups, adjusted the ~/.bashrc for the new uid, rebooted, and that account works showing the correct journalctl output.
- User account in question appears to be an sss user.
- Confirmed the user's d-bus channel is running with
ps, I see the user is running/usr/lib/systemd/systemd --user - I am able to run other
systmectl --usercommands as the user in question. I can check status, start, stop, restart the user service. - When I run
journalctl --user, here is the output:$ journalctl --user No journal files were found. -- No entries -- - When I run
journalctl --user --verify, here is the output:$ journalctl --user --verify No journal files were found. - I did all the above on another box where the sss user is also built, problem follows the user account.
- I found journalctl output! but its in /var/log/messages (the default location for journalctl output to syslog for rhel systems). Wonder why it shows there but not for the --user area?
- I checked
systemctl status user.slice --no-pagerand see the correct user slices for each uid with the correct services,systmectl --user status myserviceshows the correct slice as well. However if I checkls /var/log/journal/<number>I do not see theuser-<UID>@UUID.journallisted for the user in question.
This problem appears to only happen for the user I am testing with which was built with sss. I am not able to remove or rebuild this user as I do not manage sss on this system.
Why does this specific sss user journalctl --user output not appear but local user's do? What am I missing here? Not sure how else to troubleshoot this issue, any recommendations?
EDIT: well it does not seem to be sss related. I deployed my own AD server, integrated kerberos and SSSd, built a user, and journalctl works fine for them.
At this point I have no idea why this user's journalctl will not work, any recommendations for troubleshooting would be helpful!
EDIT2: I found journalctl output for the --user service does show in /var/log/messages but this can only be accessed by root.
