Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It only takes a minute to sign up.
Sign up to join this community
So you configured your unprivileged LXC guest, by defining
lxc.id_map = u 0 1000000000 10000
lxc.id_map = g 0 1000000000 10000
and of course assigning those subordinate UID/GID ranges to an existing user (usermod --add-sub-uids ...).
However, whenever you ssh host you get:
Read from socket failed: Connection reset by peer
However, inside the guest you can clearly see that the sshd is running (e.g. with lsof -i :22).
What could possibly be wrong?
First of all I refer you to this short troubleshooting guide for sshd which I am using as a recipe time and time again.
Only difference in this case, I used lxc-console to attach to the guest, logged in and stopped the running sshd and then started my instance on the default port 22. And then I connected from the host to the guest with heightened verbosity:
$ ssh -vvvvvvvv host.lan
OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug1: Reading configuration data /home/joe/.ssh/config
debug1: /home/joe/.ssh/config line 2: Applying options for *
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to host.lan [10.0.3.223] port 22.
debug1: Connection established.
debug1: identity file /home/joe/.ssh/id_rsa type -1
debug1: identity file /home/joe/.ssh/id_rsa-cert type -1
debug1: identity file /home/joe/.ssh/id_dsa type -1
debug1: identity file /home/joe/.ssh/id_dsa-cert type -1
debug1: identity file /home/joe/.ssh/id_ecdsa type -1
debug1: identity file /home/joe/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/joe/.ssh/id_ed25519 type -1
debug1: identity file /home/joe/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "host.lan" from file "/home/joe/.ssh/known_hosts"
debug3: load_hostkeys: found key type ED25519 in file /home/joe/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: [email protected],ssh-ed25519
debug1: SSH2_MSG_KEXINIT sent
Read from socket failed: Connection reset by peer
Hmm, there's nothing enlightening in that output. Let's check the server side output of our connection attempt:
# $(which sshd) -Dddddddp22
debug2: load_server_config: filename /etc/ssh/sshd_config
debug2: load_server_config: done config len = 724
debug2: parse_server_config: config /etc/ssh/sshd_config len 724
debug3: /etc/ssh/sshd_config:5 setting Port 22
debug3: /etc/ssh/sshd_config:9 setting Protocol 2
debug3: /etc/ssh/sshd_config:11 setting HostKey /etc/ssh/ssh_host_rsa_key
debug3: /etc/ssh/sshd_config:12 setting HostKey /etc/ssh/ssh_host_dsa_key
debug3: /etc/ssh/sshd_config:13 setting HostKey /etc/ssh/ssh_host_ed25519_key
debug3: /etc/ssh/sshd_config:15 setting UsePrivilegeSeparation yes
debug3: /etc/ssh/sshd_config:18 setting KeyRegenerationInterval 3600
debug3: /etc/ssh/sshd_config:19 setting ServerKeyBits 1024
debug3: /etc/ssh/sshd_config:22 setting SyslogFacility AUTH
debug3: /etc/ssh/sshd_config:23 setting LogLevel INFO
debug3: /etc/ssh/sshd_config:26 setting LoginGraceTime 120
debug3: /etc/ssh/sshd_config:27 setting PermitRootLogin without-password
debug3: /etc/ssh/sshd_config:28 setting StrictModes yes
debug3: /etc/ssh/sshd_config:30 setting RSAAuthentication yes
debug3: /etc/ssh/sshd_config:31 setting PubkeyAuthentication yes
debug3: /etc/ssh/sshd_config:35 setting IgnoreRhosts yes
debug3: /etc/ssh/sshd_config:37 setting RhostsRSAAuthentication no
debug3: /etc/ssh/sshd_config:39 setting HostbasedAuthentication no
debug3: /etc/ssh/sshd_config:44 setting PermitEmptyPasswords no
debug3: /etc/ssh/sshd_config:48 setting ChallengeResponseAuthentication no
debug3: /etc/ssh/sshd_config:51 setting PasswordAuthentication no
debug3: /etc/ssh/sshd_config:63 setting X11Forwarding yes
debug3: /etc/ssh/sshd_config:64 setting X11DisplayOffset 10
debug3: /etc/ssh/sshd_config:65 setting PrintMotd no
debug3: /etc/ssh/sshd_config:66 setting PrintLastLog yes
debug3: /etc/ssh/sshd_config:67 setting TCPKeepAlive yes
debug3: /etc/ssh/sshd_config:74 setting AcceptEnv LANG LC_*
debug3: /etc/ssh/sshd_config:76 setting Subsystem sftp /usr/lib/openssh/sftp-server
debug3: /etc/ssh/sshd_config:87 setting UsePAM yes
debug1: sshd version OpenSSH_6.6.1, OpenSSL 1.0.1f 6 Jan 2014
debug3: Incorrect RSA1 identifier
debug3: Incorrect RSA1 identifier
debug3: Could not load "/etc/ssh/ssh_host_rsa_key" as a RSA1 public key
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug3: Incorrect RSA1 identifier
debug3: Could not load "/etc/ssh/ssh_host_dsa_key" as a RSA1 public key
debug1: private host key: #0 type 1 RSA
debug3: Incorrect RSA1 identifier
debug3: Incorrect RSA1 identifier
debug3: Could not load "/etc/ssh/ssh_host_dsa_key" as a RSA1 public key
debug1: private host key: #1 type 2 DSA
debug3: Incorrect RSA1 identifier
debug3: Incorrect RSA1 identifier
debug3: Could not load "/etc/ssh/ssh_host_ed25519_key" as a RSA1 public key
debug1: private host key: #2 type 4 ED25519
debug1: rexec_argv[0]='/usr/sbin/sshd'
debug1: rexec_argv[1]='-Dddddddp22'
debug3: oom_adjust_setup
Set /proc/self/oom_score_adj from 0 to -1000
debug2: fd 3 setting O_NONBLOCK
debug1: Bind to port 22 on 0.0.0.0.
Server listening on 0.0.0.0 port 22.
debug2: fd 4 setting O_NONBLOCK
debug3: sock_set_v6only: set socket 4 IPV6_V6ONLY
debug1: Bind to port 22 on ::.
Server listening on :: port 22.
debug3: fd 5 is not O_NONBLOCK
debug1: Server will not fork when running in debugging mode.
debug3: send_rexec_state: entering fd = 8 config len 724
debug3: ssh_msg_send: type 0
debug3: send_rexec_state: done
debug1: rexec start in 5 out 5 newsock 5 pipe -1 sock 8
debug1: inetd sockets after dupping: 3, 3
Connection from 10.0.3.1 port 51448 on 10.0.3.223 port 22
debug1: Client protocol version 2.0; client software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1p1 Ubuntu-2ubuntu2
debug2: fd 3 setting O_NONBLOCK
debug2: Network child is on pid 558
debug3: preauth child monitor started
debug3: privsep user:group 101:65534 [preauth]
setgroups: Invalid argument [preauth]
debug1: do_cleanup [preauth]
debug3: PAM: sshpam_thread_cleanup entering [preauth]
debug1: monitor_read_log: child log fd closed
debug3: mm_request_receive entering
debug1: do_cleanup
debug3: PAM: sshpam_thread_cleanup entering
debug1: Killing privsep child 558
Pay special attention to the following the following lines from the above output:
debug1: Killing privsep child 558
indicating some issue with the privilege separation feature of OpenSSH (configuration directive UsePrivilegeSeparation yes), and:
debug3: privsep user:group 101:65534 [preauth]
setgroups: Invalid argument [preauth]
indicating that an attempt was made to change the effective GID of the process to 65534.
Now have a look again at the stanzas from the container configuration file:
lxc.id_map = u 0 1000000000 10000
lxc.id_map = g 0 1000000000 10000
which tells LXC to create a user namespace (userns) with 10000 IDs for both, group and user IDs respectively, starting at 1000000000. Inside that namespace, the UID 1000000000 becomes 0, i.e. superuser.
There are effectively two solutions to the problem:
UsePrivilegeSeparation no in sshd_configThe script container-userns-convert which is hosted on launchpad (checkout with bzr branch lp:~serge-hallyn/+junk/nsexec) and written by Serge Hallyn, one of the important contributors to LXC, and uses uidmapshift from the same repository, will assign only 10k subordinate IDs for the mapping by default.
This tripped me up. Normally I assign a block of 100000 IDs (as it's easier to read) or 65535 myself.
