0

I'm launching a script from a Jenkins server (on RHEL6) that, among other things, uses SCP with "BatchMode yes" to copy a file from a remote machine. The script runs properly outside of Jenkins, but fails inside. The verbose SCP log shows:

debug1: Next authentication method: publickey
debug1: Offering public key: /var/lib/jenkins/.ssh/id_rsa
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa

So there's something missing from the Jenkins user that is required to login. It's not a known_hosts entry, or at least the correct host is listed in /var/lib/jenkins/.ssh/known_hosts. What else could it be?

Edit: Per request, the command was

scp -vvv -o "BatchMode yes" myuser@myserver.com:myfile.txt .

Here is a more extensive log snippet:

Executing: program /usr/bin/ssh host myserver.com, user myuser, command scp -v -f myfile.txt
OpenSSH_5.3p1, OpenSSL 1.0.1e-fips 11 Feb 2013
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to myserver.com [xxx.xxx.xxx.xxx] port 22.
debug1: Connection established.
debug1: identity file /var/lib/jenkins/.ssh/identity type -1
debug1: identity file /var/lib/jenkins/.ssh/identity-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.2
debug1: match: OpenSSH_6.2 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.3
debug2: fd 3 setting O_NONBLOCK
debug1: SSH2_MSG_KEXINIT sent
debug3: Wrote 960 bytes for a total of 981
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss,ecdsa-sha2-nistp256
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug3: Wrote 24 bytes for a total of 1005
debug2: dh_gen_key: priv key bits set: 131/256
debug2: bits set: 773/1536
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug3: Wrote 208 bytes for a total of 1213
debug3: check_host_in_hostfile: host myserver.com filename /var/lib/jenkins/.ssh/known_hosts
debug3: check_host_in_hostfile: host myserver.com filename /var/lib/jenkins/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 2
debug3: check_host_in_hostfile: host xxx.xxx.xxx.xxx filename /var/lib/jenkins/.ssh/known_hosts
debug3: check_host_in_hostfile: host xxx.xxx.xxx.xxx filename /var/lib/jenkins/.ssh/known_hosts
debug3: check_host_in_hostfile: match line 3
debug1: Host 'myserver.com' is known and matches the RSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:2
debug2: bits set: 759/1536
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: Wrote 16 bytes for a total of 1229
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug3: Wrote 48 bytes for a total of 1277
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /var/lib/jenkins/.ssh/identity ((nil))
debug2: key: /var/lib/jenkins/.ssh/id_rsa (0x7f38a83ee310)
debug2: key: /var/lib/jenkins/.ssh/id_dsa ((nil))
debug2: key: /var/lib/jenkins/.ssh/id_ecdsa ((nil))
debug3: Wrote 64 bytes for a total of 1341
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey
debug3: authmethod_lookup publickey
debug3: remaining preferred: ,gssapi-with-mic,publickey
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /var/lib/jenkins/.ssh/identity
debug3: no such identity: /var/lib/jenkins/.ssh/identity
debug1: Offering public key: /var/lib/jenkins/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1709
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: SHA1 fp 72:a5:45:d3:f2:6d:15:c4:2e:f9:37:34:44:10:2b:b9:59:ee:18:c0
debug3: sign_and_send_pubkey: RSA 72:a5:45:d3:f2:6d:15:c4:2e:f9:37:34:44:10:2b:b9:59:ee:18:c0
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,keyboard-interactive).
  • you are missing from your question the whole log with -vvv options and the actual command you are running. It can be many things. – Jakuje Jan 7 '16 at 23:17
  • It would be very helpful if you could tell me some things it could be. – Ben Fulton Jan 8 '16 at 14:21
  • Server is not accepting /var/lib/jenkins/.ssh/id_rsa key for user myuser. Is the public key available on server in authorized_keys? What in in the /var/log/secure related to this login on server? – Jakuje Jan 8 '16 at 14:25
0

Here is the bit in the log that tells you what is happening:

debug1: Trying private key: /var/lib/jenkins/.ssh/identity
debug3: no such identity: /var/lib/jenkins/.ssh/identity
debug1: Offering public key: /var/lib/jenkins/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug3: Wrote 368 bytes for a total of 1709
debug1: Server accepts key: pkalg ssh-rsa blen 277
debug2: input_userauth_pk_ok: SHA1 fp 72:a5:45:d3:f2:6d:15:c4:2e:f9:37:34:44:10:2b:b9:59:ee:18:c0
debug3: sign_and_send_pubkey: RSA 72:a5:45:d3:f2:6d:15:c4:2e:f9:37:34:44:10:2b:b9:59:ee:18:c0
debug1: Trying private key: /var/lib/jenkins/.ssh/id_dsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_dsa
debug1: Trying private key: /var/lib/jenkins/.ssh/id_ecdsa
debug3: no such identity: /var/lib/jenkins/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey,keyboard-interactive).

I see a few things bad happening here.

First is that it is offering up /var/lib/jenkins/.ssh/id_rsa as a public key, when that should be a private key. It might be enlightening to take a look at your sshd configuration file.

Next it goes looking for private keys and cannot find them in any of these places:

/var/lib/jenkins/.ssh/id_dsa
/var/lib/jenkins/.ssh/id_ecdsa

Which is weird because earlier in the logfile it claimed that those files existed:

debug1: identity file /var/lib/jenkins/.ssh/identity type -1
debug1: identity file /var/lib/jenkins/.ssh/identity-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_dsa type -1
debug1: identity file /var/lib/jenkins/.ssh/id_dsa-cert type -1
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa type -1
debug1: identity file /var/lib/jenkins/.ssh/id_ecdsa-cert type -1

Here are the things I would try:

  • chmod go-rwx /var/lib/jenkins/.ssh/*
    This will ensure that only the user has read permissions to the key files. This is important as OpenSSH, if configured to be secure, will refuse to work with publicly readable key files.
  • Was the keyfile created with a password? Password protected key files will not work for automation, so ensure that the key files are not password protected.
  • Is the keyfile installed on myserver.com in /home/myuser/.ssh/authorized_keys and have you run    chmod go-rwx /home/myuser/.ssh/*    on myserver.com?
0

After a while I realized I had created the keys myself, with just "sudo ssh-keygen". This meant that the key actually belonged to the root user, and the jenkins user did not have permission to read it. If ssh had told me this, no problem. Apparently, though, the first thing ssh does if it can't read the key is to ask for a passphrase. But since it was running in batch mode, it wasn't able to ask for a passphrase, so it just gave up and failed the key. After I chown'd the key files to the jenkins user it all worked.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.