Intermittent ssh_exchange_identification error using rsync in backup script, why?

Question

I have a backup system which uses rdiff-backup locally, then copies the backup to a remote server using rsync. My rsync command looks like this:

rsync -aP --delete-after /mnt/reos-storage-2/data_upload_backup/ [email protected]:/mnt/reos-storage-2/data_upload_backup_ed-mh-pi01

This occasionally results in the following error, and my backup is not copied to the remote machine.

ssh_exchange_identification: Connection closed by remote host
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]

I have investigated the ssh_exchange_identification error and seen that it can be due to a few reasons, the main one being too many ssh connections. However, at the time this runs there should be no other ssh connections active on either machine, or maybe one or two others at most. The 'source' machine in this case is a raspberry pi running Raspian, while the 'destination' server is Ubuntu 20.04. I wondered if maybe rsync creates lots of connections?

How can I debug this? What are the possible reasons for the error?

Answer 1

Since the error message says "Connection closed by remote host", you should look at the remote host's logs: there should be a log message from sshd describing exactly why it decided to terminate the connection. Since authentication did not complete yet, sshd will not report the termination reason to the client, which is at this point still unauthenticated and might be malicious. It would not be very smart to give a possible attacker any advice on how to get through your security, would it?

An overload of malicious ssh login attempts can also (depending on kernel and sshd versions, and the sources of entropy/randomness available to the system) cause problems, often by draining the kernel's entropy pool, making the system unable to generate cryptographically secure random numbers for session keys fast enough.