0

I am pretty limited on actual server details and configurations I can share but will try my best to describe the issues and what I want to be able to do.

the issue I am facing is as follows, I have an SFTP server running Centos 7 and using the default sshd service for sftp, all of which is behind a hardware firewall, this server has been working without problem for well over 2 years now but recently connections have been dropping. I believe the issues are not directly related to the server at all, but I need to prove that to the team or at least confirm whether my belief is founded or not.

an example of how these drops appear is this. a client makes 3 separate connections all at the same time the first connection succeeds and a file is uploaded while the other 2 connections fail. in the hardware firewall, the logging shows all 3 connections open and close but the last 2 connections have the message that states the server reset the connection. on the server when I view the /var/logs/secure and /var/logs/messages log files I find records for the first successful connection but have absolutely nothing for the 2 failed.

the server does have fail2ban which I have also checked and nothing is being blocked there, in fact, this issue is happening over multiple IP addresses and some of those addresses are set to be ignored by fail2ban altogether.

the server does have SELinux enabled and auditd enabled but logs are harder for me to read and are quite noisy (if there is a clean way to review these logs please let me know)

the issue is sporadic happening multiple times in an hour then no issues at all for 2-3 hours.

I have quite a number of these servers setup for different clients and when I have come across these types of issues in the past, reviewing these logs tend to be all I need to do but this time I have 2 separate system (firewall and Server) giving me conflicting statements and now I am not sure how to proceed.

is there any additional logging I can look at or even something I could install to monitor network traffic and confirm either way if this failed traffic is getting to my server?

Answer

using tshark to monitor net traffic worked well

using the below command I Filitered traffic to host I know was experiencing the issue and saved the capture.

tshark -i eth0 -f "host x.x.x.x" -w "capture.pcap" -F pcap

once I got a report the issue occurred again I stopped the capture and transfer the resulting file to my desktop to open in wireshark. the resulting capture showed the server was resetting the connection as you can see in the example capture below on the last line where the server issues a [RST, ACK] (reset, Acknowledgement)

2467    4501.231127 <CLEINT_IP> <SERVER_IP> TCP 66  58274 → 22 [SYN, ECE, CWR] Seq=0 Win=65535 Len=0 MSS=1460 WS=128
2468    4501.231178 <SERVER_IP> <CLEINT_IP> TCP 62  22 → 58274 [SYN, ACK] Seq=0 Ack=1 Win=29200 Len=0 MSS=1460 WS=128
2469    4501.231646 <CLEINT_IP> <SERVER_IP> TCP 60  58274 → 22 [ACK] Seq=1 Ack=1 Win=4194304 Len=0
2470    4501.231737 <CLEINT_IP> <SERVER_IP> SSHv2   85  Client: Protocol (SSH-2.0-WinSCP_release_5.15.3)
2471    4501.231750 <SERVER_IP> <CLEINT_IP> TCP 54  22 → 58274 [ACK] Seq=1 Ack=32 Win=29312 Len=0
2472    4501.232298 <SERVER_IP> <CLEINT_IP> TCP 54  22 → 58274 [RST, ACK] Seq=1 Ack=32 Win=29312 Len=0

1 Answer 1

1

A question that needs investigating is whether the firewall acts as the ssh server endpoint for your client, or whether it transparently passes traffic through to the real server.

Run tshark (or the graphical wireshark on the server, and look for connections initiated from the client and/or firewall, particularly with reference to the failing sessions.

If you're not seeing any traffic inbound for these failing sessions then the firewall is not passing them through and it is the point of failure. If you are, then it could be your server that's at fault.

2
  • 1
    Thanks, Roaima, I completely forgot about tshark. I installed that and had it run until another drop was reported. to my dismay, it does hit the server and I can see an RST ACK being issued by the server. I now need to work out why it is doing it
    – Balasarius
    Apr 12, 2023 at 9:49
  • You may see something reported in one of kern.log (bad news), auth.log (authentication failure, so IP address restrictions), or daemon.log (anything else), @Balasarius Apr 12, 2023 at 11:35

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .