I got a Fedora 23 Server edition installation which I have done pretty much nothing with admin-wise besides setting up
samba, and users.
The other day I was completely locked out of my system with the message
ssh_exchange_identification: read: Connection reset by peer which with some googling led me to ssh_exchange_identification: read: Connection reset by peer
So I assumed that
fail2ban had mucked up with the banning. No biggie, just log in to the machine 'physically' and reset them.
Upon trying to do this I see my console gets flooded with a log message (which I can't remember, "systemd.service something something read-only something"). This flooding made it impossible to log in. I started typing my credentials, only to have it broken because a log-message would be printed and reset the credentials.
I was forced to a hard-reboot, ie. press reset-button / hold power-button on the machine.
As I want to get to the bottom of the problem and find out what the trouble was I did some log-digging. I don't seem to be able to find the cause however.
If I do
journalctl --since 2016-04-10 I get the following
Apr 10 12:19:32 Server smartd: Device: /dev/sdc [SAT], 47 Currently unreadable (pending) sectors -- Reboot -- Apr 11 20:27:53 Server systemd-journal: Runtime journal is using 8.0M (max allowed 81.0M, trying to leave 121.5M free of 802.5M available → current limit 81.0M).
I dropped all logs from, when I assume the trobles started, till I did the hard-reboot. (The
/dev/sdc drive has been giving that error since I got it)
I also read on https://freedesktop.org/wiki/Software/systemd/Debugging/ that you should check
/var/log/messages for logs.
The relevant parts here that I can see are
Mar 6 18:52:49 Server audit: USER_AUTH pid=3435 uid=0 auid=4294967295 ses=4294967295 subj=system_u:system_r:sshd_t:s0-s0:c0.c1023 msg='op=PAM:authentication grantors=? acct="root" exe="/usr/sbin/sshd" hostname=22.214.171.124 addr=126.96.36.199 terminal=ssh res=failed' Apr 11 20:28:46 Server rsyslogd-2177: imjournal: begin to drop messages due to rate-limiting Apr 11 20:39:15 Server rsyslogd-2177: imjournal: 91404 messages lost due to rate-limiting Apr 11 20:39:15 Server dnf: repo: using cache for: rpmfusion-free
Which tells me that there were 91404 log messages. I can't find a single one of them however. All were dropped as far as I can tell.
Since I want to know what actually 'broke' my system, is there anywhere else that the logs might be stored? Or do I just have to pray that the stars don't align to cause the same problem again?