3

I'm running a server at home with Ubuntu 14.04. I have two users on the machine, say user1 and user2, each able to ssh into the box. While user1 logs in very quickly, user2 takes about 20 seconds to log in. The same behavior happens regardless of which user is logging in from any particular machine.

Logging in as user2 with ssh -v shows that the main pause happens after this message:

debug1: Sending environment.

The .bashrc files for the two users are identical. (A diff on them produces no output.) Same for the .profile files for user1 and user2.

On the server, there is a line in /etc/ssh/sshd_config with AllowUsers user1 user2. That same file has UseDNS no in it already. (I only mention that because most pages I found while searching for a solution mentioned to add that line.)

This issue started only about a week ago; I've had no such issues for the previous couple of years that I've been running the machine.

On a whim, I created a new user, user3, gave that user ssh access by editing sshd_config appropriately, and log in for the new user is fast. So it really seems to be something about the way user2 is configured. But what?

Where should I look to find the root cause of the delay for the log in for user2?

EDIT: Below are the results from tail -f /var/log/auth.log when user2 logs in:

Jun 21 11:27:10 topo sshd[18386]: Connection from re.mo.te.ip port XXXXX on se.rv.er.ip port XX
Jun 21 11:27:21 topo sshd[18386]: Accepted password for user2 from re.mo.te.ip port XXXXX ssh2
Jun 21 11:27:21 topo sshd[18386]: pam_unix(sshd:session): session opened for user user2 by (uid=0)
Jun 21 11:27:21 topo systemd-logind[3347]: New session 25 of user user2.
Jun 21 11:28:03 topo sshd[18386]: User child is on pid XXXXX
Jun 21 11:28:03 topo sshd[18455]: Starting session: shell on pts/6 for user2 from re.mo.te.ip port XXXXX

Note the time stamps. The delay occurs after the last message here. No messages appear in auth.log once user2 is finally logged in.

EDIT: Below are some results from strace -f -p <sshd pid> -o out.tx -tt where there are delays.

22733 12:59:10.414388 open("/usr/lib/python2.7/dist-packages/twisted/python/compat.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:10.840303 open("/usr/lib/python2.7/dist-packages/twisted/python/compat.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:11.264716 open("/usr/lib/python2.7/dist-packages/twisted/python/compatmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:11.717471 open("/usr/lib/python2.7/dist-packages/twisted/python/compat.py", O_RDONLY) = 4
22733 12:59:12.660275 fstat(4, {st_mode=S_IFREG|0644, st_size=11146, ...}) = 0
...
22733 12:59:12.661502 open("/usr/lib/python2.7/dist-packages/twisted/python/__future__.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:13.212844 open("/usr/lib/python2.7/dist-packages/twisted/python/__future__.so", O_RDONLY) = -1 ENOENT (No such file or directory)
...
22733 12:59:14.441963 open("/usr/bin/dis.py", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:14.900322 open("/usr/bin/dis.pyc", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:15.401750 stat("/usr/lib/python2.7/dis", 0x7ffcb599ddd0) = -1 ENOENT (No such file or directory)
...
22733 12:59:17.315288 open("/usr/lib/python2.7/dist-packages/twisted/internet/termios.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:17.967173 open("/usr/lib/python2.7/dist-packages/twisted/internet/termiosmodule.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:18.511526 open("/usr/lib/python2.7/dist-packages/twisted/internet/termios.py", O_RDONLY) = -1 ENOENT (No such file or directory)
...
22733 12:59:25.993287 open("/usr/lib/python2.7/dist-packages/landscape/monitor/time.x86_64-linux-gnu.so", O_RDONLY) = -1 ENOENT (No such file or directory)
22733 12:59:26.612018 open("/usr/lib/python2.7/dist-packages/landscape/monitor/time.so", O_RDONLY) = -1 ENOENT (No such file or directory)
...
22733 12:59:29.688166 open("/proc/37/stat", O_RDONLY) = 9
22733 12:59:30.239695 fstat(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
...
22733 12:59:30.240450 open("/proc/uptime", O_RDONLY) = 10
22733 12:59:31.634404 fstat(10, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
...
22733 12:59:34.287758 open("/proc/557/stat", O_RDONLY) = 9
22733 12:59:34.851662 fstat(9, {st_mode=S_IFREG|0444, st_size=0, ...}) = 0
...
22758 12:59:39.500397 open(".", O_RDONLY) = 3
22758 12:59:40.377193 fchdir(3)         = 0
22758 12:59:40.377312 newfstatat(AT_FDCWD, "/var/lib/apt//lists/", {st_mode=S_IFDIR|0755, st_size=12288, ...}, AT_SYMLINK_NOFOLLOW) = 0
22758 12:59:40.377378 open(".", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_DIRECTORY|O_NOFOLLOW) = 4
22758 12:59:40.878676 fchdir(4)         = 0
...
22759 12:59:41.680427 open("/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
22759 12:59:42.249796 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0P \2\0\0\0\0\0"..., 832) = 832
...
22759 12:59:46.019309 open("/etc/lsb-release", O_RDONLY|O_CLOEXEC) = 64
22759 12:59:46.948115 fstat(64, {st_mode=S_IFREG|0644, st_size=105, ...}) = 0
...
22786 12:59:50.251281 open("/dev/null", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
22786 12:59:50.970074 fcntl(1, F_GETFD) = 0
...
22785 12:59:53.338632 select(10, [3 4 6], [], NULL, NULL) = 1 (in [3])
22785 12:59:53.771002 rt_sigprocmask(SIG_BLOCK, [CHLD], [], 8) = 0

A lot of this seems related to python libraries. However, they are also loaded when user1 logs in, just without these 0.5s pauses.

  • Try tail -f /var/log/auth.log on the server while sshing into the machine as user2`. Maybe something useful is logged there? – Martin von Wittich Jun 21 '16 at 18:24
  • Do you have a remote authentication server (radius or tacacs etc.) configured in you server? – coffeMug Jun 21 '16 at 18:31
  • @MartinvonWittich I edited to include that information. – Bill Kronholm Jun 21 '16 at 18:33
  • What exactly is bash doing? E.g. compare output between users for ssh -t host 'bash -xvl' or so – thrig Jun 21 '16 at 18:34
  • instead of -v switch, user -vvv for a more detailed verbose output while logging in as user2. It looks like there is a 11 second delay before the password is accepted, compared to connection establishing. Check user2's .ssh directory permissions and the permissions for files inside .ssh. It might be trying to authenticate using a key pair first. – MelBurslan Jun 21 '16 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.