SCP sudden abort: broken pipe, message authentication code incorrect

Question

Related topics

My problem is similar to but not exactly the same as SSH broken pipe, message authentication code incorrect for which there is no answer.

Task

Copy large files from one Linux to another. Both stand at the same ISP location.

Setup

Source and target are both: Ubuntu 16.04.3 LTS

SSH version on both: OpenSSH_7.2p2 Ubuntu-4ubuntu2.2, OpenSSL 1.0.2g 1 Mar 2016

The source machine has been in use for a year, no problems. The target machine is a freshly set up dedicated server (1 day).

scp command:

scp -P [customport] /some/large/file user@targetmachine:/target/folder/

The file is about 20 GB in size.

Problem description

It usually aborts after about 3-4%. Full speed is at about 112MB/s. When I throttle with eg scp -l 16384 it goes at about 2MB/s, aborts much later in time, but at a similar percentage.

The abort is always in the exact same way. The client gets:

Write failed: Broken pipe 
lost connection

While the server has this in /var/log/auth.log

Nov 24 13:04:54 Ubuntu-1604-xenial-64-minimal-no-hwe sshd[1900]: Corrupted MAC on input.
Nov 24 13:04:54 Ubuntu-1604-xenial-64-minimal-no-hwe sshd[1900]: fatal: ssh_dispatch_run_fatal: Connection from [client-ip] port 54050: message authentication code incorrect

Investigation

I have tried both with iptables enabled and disabled, no change.

Out of about 10 attempts, 1 succeeded to the end, then the next file aborted again.

It seems that after rebooting the target machine, more bytes can be written to it.

SSH is no problem. I can keep an idle ssh connection open for hours, or one where the top command is running and it does not break.

Questions

This is a blocker. First, it seems impossible to copy a 200gb file over. Second, I don't want a machine in production with networking issues.

What can I do to further investigate this?

I've read elsewhere that it might be a network card/hardware issue, how can I prove this to my provider to get a replacement?

Update 1

The result for a 10 minute mtr looks good:

└─(~)─(49 files, 12Gb)─> mtr -r -c 600 -rw [targetserver]
Start: Fri Nov 24 18:36:21 2017
HOST: Ubuntu-1404-trusty-64-minimal             Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- static.XX.XX.XX.XX.clients.your-server.de  0.0%   600    0.5   0.3   0.2  24.5   1.3
  2.|-- core24.fsn1.hetzner.com                    0.0%   600    0.3   0.3   0.2   6.8   0.4
  3.|-- core22.fsn1.hetzner.com                    0.0%   600    0.4   0.4   0.3   9.7   0.8
  4.|-- ex9k2.dc1.fsn1.hetzner.com                 0.0%   600    0.4   0.5   0.3   6.8   0.8
  5.|-- my.target.hostname                         0.0%   600    0.4   0.3   0.3   0.4   0.0
┌(myuser@Ubuntu-1404-trusty-64-minimal)─(✓)─(06:46 PM Fri Nov 24)

Right after that I tried another scp, it failed at 44% after 7.5GB, rate was 111MB/sec. The failure again came instantly, no stalling before that.

Regarding the possible duplicate: I always got the "broken pipe", never the "Protocol wrong type for socket". Not using Mac, both Linux (versions above). Not using rsync. The answer there was that the user has put another network card into the server, without figuring out what the actual cause was, as far as I understand. I don't have this option (dedicated server in remote host center).

Here's the output of lshw regarding the network card:

myuser@Ubuntu-1604-xenial-64-minimal-no-hwe /home/myuser # lshw -class network
  *-network:0 DISABLED
       description: Ethernet interface
       product: NetXtreme II BCM57810 10 Gigabit Ethernet
       vendor: Broadcom Corporation
       physical id: 0
       bus info: pci@0000:61:00.0
       logical name: eth0
       version: 10
       serial: e0:d5:5e:1e:73:18
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=bnx2x driverversion=1.712.30-0 firmware=bc 7.14.2 latency=0 link=no multicast=yes port=fibre
       resources: iomemory:14c0-14bf iomemory:14c0-14bf iomemory:14c0-14bf irq:81 memory:14c0b000000-14c0b7fffff memory:14c0a800000-14c0affffff memory:14c0b810000-14c0b81ffff memory:e5f80000-e5ffffff memory:14c0ba20000-14c0bc1ffff memory:14c0bca0000-14c0bd1ffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: NetXtreme II BCM57810 10 Gigabit Ethernet
       vendor: Broadcom Corporation
       physical id: 0.1
       bus info: pci@0000:61:00.1
       logical name: eth1
       version: 10
       serial: e0:d5:5e:1e:73:1a
       capacity: 1Gbit/s
       width: 64 bits
       clock: 33MHz
       capabilities: pm vpd msix pciexpress bus_master cap_list rom ethernet physical fibre 1000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=bnx2x driverversion=1.712.30-0 firmware=bc 7.14.2 latency=0 link=no multicast=yes port=fibre
       resources: iomemory:14c0-14bf iomemory:14c0-14bf iomemory:14c0-14bf irq:102 memory:14c0a000000-14c0a7fffff memory:14c09800000-14c09ffffff memory:14c0b800000-14c0b80ffff memory:e5f00000-e5f7ffff memory:14c0b820000-14c0ba1ffff memory:14c0bc20000-14c0bc9ffff
  *-network:0
       description: Ethernet interface
       product: I350 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:62:00.0
       logical name: eth2
       version: 01
       serial: 6c:b3:11:23:32:18
       size: 1Gbit/s
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list rom ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.3.0-k duplex=full firmware=1.63, 0x80000cbb ip=94.130.51.145 latency=0 link=yes multicast=yes port=twisted pair speed=1Gbit/s
       resources: irq:71 memory:e5900000-e59fffff memory:e5a84000-e5a87fff memory:e5a00000-e5a7ffff memory:14c0bf60000-14c0bf7ffff memory:14c0bf40000-14c0bf5ffff
  *-network:1 DISABLED
       description: Ethernet interface
       product: I350 Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:62:00.1
       logical name: eth3
       version: 01
       serial: 6c:b3:11:23:32:19
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=igb driverversion=5.3.0-k firmware=1.63, 0x80000cbb latency=0 link=no multicast=yes port=twisted pair
       resources: irq:82 memory:e5800000-e58fffff memory:e5a80000-e5a83fff memory:14c0bf20000-14c0bf3ffff memory:14c0bf00000-14c0bf1ffff
  *-network DISABLED
       description: Ethernet interface
       physical id: 1
       logical name: virbr0-nic
       serial: 52:54:00:80:b4:28
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=no multicast=yes port=twisted pair speed=10Mbit/s

This reminds me, I did install KVM

apt-get install qemu-kvm libvirt-bin ubuntu-vm-builder bridge-utils

But no VM is on yet.

Answer 1

I've had a similar problem when either using scp or rsync + samba/cifs.

The problem was resolved on the rsync + samba/cifs side by bypassing the write cache using --cache=none when mounting the server on the client (see also rsync keeps disconnecting: broken pipe). An in depth explanation regarding the root cause of this problem is given at Make Linux write to network filesystem concurrently with local disk reads.

With scp you could consider throttling the transfer rate to avoid filling the page cache before the disk has been able to catch up, see for example https://stackoverflow.com/questions/30020519/broken-pipe-error-on-scp.

Answer 2

This was a "minimal-no-hwe" installation. The "minimal" version of Ubuntu would most likely have worked from the start.

These commands were used to install hwe into this malfunctioning no-hwe version (so no complete re-install):

apt-get install --install-recommends linux-generic-hwe-16.04
shutdown -r now

After this, all scp copies work, no aborts.

A side note, the greeting in the terminal still shows

"myuser@Ubuntu-1604-xenial-64-minimal-no-hwe"

even though hwe is on now.

I clarify one more time the behavior before this fix: All large scp TO this machine, from various locations, aborted, while all scp FROM this machine, to various locations, succeeded.

This is the server spec https://www.hetzner.de/epyc-server although the hoster does not specify the models for mainboard/networking.