1

Why are these sshd-pts processes stuck in status D? It seems the system load averages are increasing due to these processes. How can I remove them without restarting my server.

(add ppid and etime)

[root@manager ~]# ps -eo pid,ppid,user,state,etime,command,wchan |grep sshd |awk -F " " '{if($4=="D")print}'
 3024  7162 root     D 31-00:45:56 sshd: root@pts/10           tty_ldisc_hangup
 3799 23740 root     D 62-03:49:15 sshd: root@pts/7            tty_ldisc_ref_wait
 4883 23740 root     D 29-02:12:59 sshd: root@pts/11           tty_ldisc_ref_wait
 7162 23740 root     D 34-21:39:42 sshd: root@pts/10           tty_ldisc_ref_wait
 8011 15566 root     D 62-21:06:45 sshd: root@pts/4            tty_ldisc_hangup
 9297 29509 root     D 71-21:44:30 sshd: root@pts/5            tty_ldisc_hangup
13927 32658 root     D 48-15:41:05 sshd: root@pts/8            tty_ldisc_hangup
14488     1 root     D 62-17:42:02 sshd: root@pts/6            tty_ldisc_ref_wait
15007 23740 root     D 47-23:40:33 sshd: root@pts/9            tty_ldisc_ref_wait
15566     1 root     D 68-22:23:34 sshd: root@pts/4            tty_ldisc_ref_wait
18017     1 root     D 82-11:50:11 sshd: root@pts/3            tty_ldisc_ref_wait
22081  4883 root     D 24-21:08:20 sshd: root@pts/11           tty_ldisc_hangup
25157 15007 root     D 41-11:34:06 sshd: root@pts/9            tty_ldisc_hangup
28168 18017 root     D 82-11:49:11 sshd: root@pts/3            tty_ldisc_hangup
29509     1 root     D 71-21:47:22 sshd: root@pts/5            tty_ldisc_ref_wait
29718  3799 root     D 61-02:02:54 sshd: root@pts/7            tty_ldisc_hangup
31394 14488 root     D 62-13:38:32 sshd: root@pts/6            tty_ldisc_hangup
32658 23740 root     D 58-21:47:54 sshd: root@pts/8            tty_ldisc_ref_wait

stack info

[root@manager ~]# cat /proc/3024/stack 
[<ffffffff81396c99>] tty_ldisc_hangup+0xc9/0x220
[<ffffffff8138e4fc>] __tty_hangup+0x30c/0x410
[<ffffffff8138e911>] tty_vhangup_self+0x21/0x50
[<ffffffff8117b043>] sys_vhangup+0x23/0x30
[<ffffffff816d8639>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

[root@manager ~]# cat /proc/3799/stack 
[<ffffffff813961f0>] tty_ldisc_ref_wait+0x20/0x50
[<ffffffff8138e8b8>] tty_poll+0x58/0x90
[<ffffffff8118f92e>] do_select+0x36e/0x680
[<ffffffff8118fe1b>] core_sys_select+0x1db/0x300
[<ffffffff8118fffa>] SyS_select+0xba/0x110
[<ffffffff816d8639>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff

(Add more info:)

After checking the ppid, I've found that the processes that stuck at tty_ldisc_hangup are children of processes that stuck at tty_ldisc_ref_wait.

e.g. process 3024 is the child of process 7162. It seems that 7162 is waiting for 3024 to hangup tty.

And according to slm's suggestion, I checked lsof of these processes. Following is the truncated output of 3024 and 7162.

(3024)

sshd    3024 root    0u   CHR    1,3      0t0       1028 /dev/null
sshd    3024 root    1u   CHR    1,3      0t0       1028 /dev/null
sshd    3024 root    2u   CHR    1,3      0t0       1028 /dev/null
sshd    3024 root    3u  sock    0,6      0t0 1037695934 protocol: TCP
sshd    3024 root    5r  FIFO    0,8      0t0 1037701548 pipe
sshd    3024 root    6w  FIFO   0,16      0t0 1037701541 /run/systemd/sessions/705007.ref
sshd    3024 root    7w  FIFO    0,8      0t0 1037701548 pipe
sshd    3024 root    9u   CHR 136,10      0t0         13 /dev/pts/10
sshd    3024 root   10r  FIFO    0,8      0t0 2320263517 pipe
sshd    3024 root   11w  FIFO    0,8      0t0 2320263517 pipe

(7162)

sshd    7162 root    0u   CHR                1,3      0t0       1028 /dev/null
sshd    7162 root    1u   CHR                1,3      0t0       1028 /dev/null
sshd    7162 root    2u   CHR                1,3      0t0       1028 /dev/null
sshd    7162 root    3u  sock                0,6      0t0 1037695934 protocol: TCP
sshd    7162 root    4u  unix 0xffff88069a94e580      0t0 1037701545 socket
sshd    7162 root    5r  FIFO                0,8      0t0 1037701548 pipe
sshd    7162 root    6w  FIFO               0,16      0t0 1037701541 /run/systemd/sessions/705007.ref
sshd    7162 root    7w  FIFO                0,8      0t0 1037701548 pipe
sshd    7162 root    8u   CHR                5,2      0t0       1644 /dev/ptmx
sshd    7162 root   12u   CHR                5,2      0t0       1644 /dev/ptmx
sshd    7162 root   13u   CHR                5,2      0t0       1644 /dev/ptmx

checking session info:

[root@manager ~]# loginctl session-status 705007
705007 - root (0)
       Since: Thur 2018-06-14 10:10:31 UTC; 1 months 4 days ago
      Leader: 7162 (sshd)
      Remote: 10.161.16.14
     Service: sshd; type tty; class user
       State: active
        Unit: session-705007.scope
              ├─3024 sshd: root@pts/10   
              └─7162 sshd: root@pts/10

And the remove ip is actually my host ip.

[root@tgdc-manager-machine ~]# hostname -I
10.161.16.14
0

Background

By definition anything in status D:

D uninterruptible sleep (usually IO)

Is waiting in a portion of the kernel's code that is not interruptible. Things such as disk I/O. This doesn't mean that you cannot send it a signal via the kill command, if just means that it won't act on it, until it's out of this portion of the kernel.

So the hint here is I/O. If these connections are truly waiting on disk I/O, I'd take a look at them using lsof -p <pid> to see if there's some path to the HDD that they're blocked on.

Triage

$ lsof -p 3499
COMMAND  PID    USER   FD   TYPE             DEVICE SIZE/OFF     NODE NAME
sshd    3499 vagrant  cwd    DIR              253,0      239       64 /
sshd    3499 vagrant  rtd    DIR              253,0      239       64 /
sshd    3499 vagrant  txt    REG              253,0   853040 33982191 /usr/sbin/sshd
sshd    3499 vagrant  mem    REG              253,0    15480 33565594 /usr/lib64/security/pam_lastlog.so
sshd    3499 vagrant  mem    REG              253,0    15632 33565575 /usr/lib64/libpam_misc.so.0.82.0
sshd    3499 vagrant  mem    REG              253,0   309280 33801011 /usr/lib64/security/pam_systemd.so
sshd    3499 vagrant  mem    REG              253,0    19600 33565595 /usr/lib64/security/pam_limits.so
sshd    3499 vagrant  mem    REG              253,0    11152 33565593 /usr/lib64/security/pam_keyinit.so
sshd    3499 vagrant  mem    REG              253,0    40784 33565922 /usr/lib64/security/pam_namespace.so
sshd    3499 vagrant  mem    REG              253,0    11200 33565598 /usr/lib64/security/pam_loginuid.so
sshd    3499 vagrant  mem    REG              253,0    19760 33565930 /usr/lib64/security/pam_selinux.so
sshd    3499 vagrant  mem    REG              253,0    44600 33565499 /usr/lib64/libcrack.so.2.9.0
sshd    3499 vagrant  mem    REG              253,0    23832 33565556 /usr/lib64/libpwquality.so.1.0.2
sshd    3499 vagrant  mem    REG              253,0    11232 33565557 /usr/lib64/security/pam_pwquality.so
sshd    3499 vagrant  mem    REG              253,0     6984 33565924 /usr/lib64/security/pam_permit.so
sshd    3499 vagrant  mem    REG              253,0    11144 33565597 /usr/lib64/security/pam_localuser.so
sshd    3499 vagrant  mem    REG              253,0    11176 33565923 /usr/lib64/security/pam_nologin.so
sshd    3499 vagrant  mem    REG              253,0     6872 33565583 /usr/lib64/security/pam_deny.so
sshd    3499 vagrant  mem    REG              253,0    15408 33565935 /usr/lib64/security/pam_succeed_if.so
sshd    3499 vagrant  mem    REG              253,0    57728 33565941 /usr/lib64/security/pam_unix.so
sshd    3499 vagrant  mem    REG              253,0    11168 33565587 /usr/lib64/security/pam_faildelay.so
sshd    3499 vagrant  mem    REG              253,0    15552 33565585 /usr/lib64/security/pam_env.so
sshd    3499 vagrant  mem    REG              253,0    15496 33565932 /usr/lib64/security/pam_sepermit.so
sshd    3499 vagrant  mem    REG              253,0    62184 33558009 /usr/lib64/libnss_files-2.17.so
sshd    3499 vagrant  mem    REG              253,0    68192 33565404 /usr/lib64/libbz2.so.1.0.6
sshd    3499 vagrant  mem    REG              253,0    99944 33565444 /usr/lib64/libelf-0.170.so
sshd    3499 vagrant  mem    REG              253,0    19896 33565252 /usr/lib64/libattr.so.1.1.0
sshd    3499 vagrant  mem    REG              253,0    15688 33565266 /usr/lib64/libkeyutils.so.1.5
sshd    3499 vagrant  mem    REG              253,0    58728 33565394 /usr/lib64/libkrb5support.so.0.1
sshd    3499 vagrant  mem    REG              253,0    11464 33557778 /usr/lib64/libfreebl3.so
sshd    3499 vagrant  mem    REG              253,0   251776 33565174 /usr/lib64/libnspr4.so
sshd    3499 vagrant  mem    REG              253,0    20040 33565177 /usr/lib64/libplc4.so
sshd    3499 vagrant  mem    REG              253,0    15744 33565178 /usr/lib64/libplds4.so
sshd    3499 vagrant  mem    REG              253,0   194944 33565179 /usr/lib64/libnssutil3.so
sshd    3499 vagrant  mem    REG              253,0  1249544 33574183 /usr/lib64/libnss3.so
sshd    3499 vagrant  mem    REG              253,0   164288 33565958 /usr/lib64/libsmime3.so
sshd    3499 vagrant  mem    REG              253,0   323664 33565980 /usr/lib64/libssl3.so
sshd    3499 vagrant  mem    REG              253,0   470360 33565368 /usr/lib64/libssl.so.1.0.2k
sshd    3499 vagrant  mem    REG              253,0   121320 33574189 /usr/lib64/libsasl2.so.3.0.0
sshd    3499 vagrant  mem    REG              253,0   144792 33565153 /usr/lib64/libpthread-2.17.so
sshd    3499 vagrant  mem    REG              253,0    88720 33554537 /usr/lib64/libgcc_s-4.8.5-20150702.so.1
sshd    3499 vagrant  mem    REG              253,0   297360 33800995 /usr/lib64/libdw-0.170.so
sshd    3499 vagrant  mem    REG              253,0    19384 33565450 /usr/lib64/libgpg-error.so.0.10.0
sshd    3499 vagrant  mem    REG              253,0   535064 33565460 /usr/lib64/libgcrypt.so.11.8.2
sshd    3499 vagrant  mem    REG              253,0    85952 33574789 /usr/lib64/liblz4.so.1.7.5
sshd    3499 vagrant  mem    REG              253,0   157424 33565398 /usr/lib64/liblzma.so.5.2.2
sshd    3499 vagrant  mem    REG              253,0    44448 33565157 /usr/lib64/librt-2.17.so
sshd    3499 vagrant  mem    REG              253,0  1139680 33557999 /usr/lib64/libm-2.17.so
sshd    3499 vagrant  mem    REG              253,0    20032 33565257 /usr/lib64/libcap.so.2.22
sshd    3499 vagrant  mem    REG              253,0   402384 33565268 /usr/lib64/libpcre.so.1.2.0
sshd    3499 vagrant  mem    REG              253,0    23968 33565452 /usr/lib64/libcap-ng.so.0.0.0
sshd    3499 vagrant  mem    REG              253,0   117680 33558001 /usr/lib64/libnsl-2.17.so
sshd    3499 vagrant  mem    REG              253,0  2173512 33557991 /usr/lib64/libc-2.17.so
sshd    3499 vagrant  mem    REG              253,0    15848 33565175 /usr/lib64/libcom_err.so.2.1
sshd    3499 vagrant  mem    REG              253,0   210840 33565386 /usr/lib64/libk5crypto.so.3.1
sshd    3499 vagrant  mem    REG              253,0   963576 33565392 /usr/lib64/libkrb5.so.3.3
sshd    3499 vagrant  mem    REG              253,0   320408 33565382 /usr/lib64/libgssapi_krb5.so.2.2
sshd    3499 vagrant  mem    REG              253,0   106848 33565155 /usr/lib64/libresolv-2.17.so
sshd    3499 vagrant  mem    REG              253,0    41080 33557995 /usr/lib64/libcrypt-2.17.so
sshd    3499 vagrant  mem    REG              253,0    90664 33565173 /usr/lib64/libz.so.1.2.7
sshd    3499 vagrant  mem    REG              253,0    14872 33565161 /usr/lib64/libutil-2.17.so
sshd    3499 vagrant  mem    REG              253,0    61952 33725051 /usr/lib64/liblber-2.4.so.2.10.7
sshd    3499 vagrant  mem    REG              253,0   352600 33725053 /usr/lib64/libldap-2.4.so.2.10.7
sshd    3499 vagrant  mem    REG              253,0    19776 33557997 /usr/lib64/libdl-2.17.so
sshd    3499 vagrant  mem    REG              253,0  2512832 33565366 /usr/lib64/libcrypto.so.1.0.2k
sshd    3499 vagrant  mem    REG              253,0   203800 33801008 /usr/lib64/libsystemd.so.0.6.0
sshd    3499 vagrant  mem    REG              253,0   155784 33565267 /usr/lib64/libselinux.so.1
sshd    3499 vagrant  mem    REG              253,0    61672 33565573 /usr/lib64/libpam.so.0.83.1
sshd    3499 vagrant  mem    REG              253,0   127096 33565455 /usr/lib64/libaudit.so.1.0.0
sshd    3499 vagrant  mem    REG              253,0    42520 33565448 /usr/lib64/libwrap.so.0.7.6
sshd    3499 vagrant  mem    REG              253,0    11344 33724998 /usr/lib64/libfipscheck.so.1.2.1
sshd    3499 vagrant  mem    REG              253,0   164240 33557984 /usr/lib64/ld-2.17.so
sshd    3499 vagrant    0u   CHR                1,3      0t0     5078 /dev/null
sshd    3499 vagrant    1u   CHR                1,3      0t0     5078 /dev/null
sshd    3499 vagrant    2u   CHR                1,3      0t0     5078 /dev/null
sshd    3499 vagrant    3u  IPv4              28862      0t0      TCP localhost.localdomain:ssh->10.0.2.2:49736 (ESTABLISHED)
sshd    3499 vagrant    4u  unix 0xffff98d19f296400      0t0    28968 socket
sshd    3499 vagrant    5u  unix 0xffff98d19f010800      0t0    28971 socket
sshd    3499 vagrant    6w  FIFO               0,19      0t0    28964 /run/systemd/sessions/13.ref
sshd    3499 vagrant    7r  FIFO                0,9      0t0    28983 pipe
sshd    3499 vagrant    8w  FIFO                0,9      0t0    28983 pipe
sshd    3499 vagrant    9u  unix 0xffff98d19a775c00      0t0    28984 /tmp/ssh-UEdXO0wvf9/agent.3499
sshd    3499 vagrant   10u   CHR                5,2      0t0     6796 /dev/ptmx
sshd    3499 vagrant   14u   CHR                5,2      0t0     6796 /dev/ptmx
sshd    3499 vagrant   15u   CHR                5,2      0t0     6796 /dev/ptmx 

disk I/O

It's entirely possible that someone may have been in an SSH session and changed directories (cd) to some filesystem that may no longer be available or is hung up, which is leading to your status D's.

network I/o

The other thought is that the blocked I/O may in fact be network connections to another server, that's blocking.

For the TCP connections that an ssh is actively using they'll look like this:

sshd    3499 vagrant    3u  IPv4              28862      0t0      TCP localhost.localdomain:ssh->10.0.2.2:49736 (ESTABLISHED)

I'd look for this and confirm that you can ping these servers, and also that you can curl -v telnet://<ip>:22 to these servers as well.

  • The wchan column in the ps listing in the original post seems to suggest that the waiting is somehow TTY-related. This question on Server Fault and this one on Stack Overflow both relate to a similar problem which was ultimately caused by a hpsa kernel module and its interaction with a buggy RAID controller firmware(!). – telcoM Jul 18 '18 at 11:43
  • @telcoM - agree it's likely pts related, that would show up in the lsof output as well. – slm Jul 18 '18 at 11:45
  • 1
    The stack trace is unequivocal. The first process is executing vhangup() and is blocked inside the line discipline code. – JdeBP Jul 18 '18 at 12:51
  • add more info and lsof output. These is a TCP connection info. But it's a sock type not IPv4. – Wenyi Li Jul 19 '18 at 8:20
  • @WenyiLi - I'm showing general approaches to the problem, not exhaustive. In multiple discussions w/ other colleagues there really isn't any way to recover from D state, you're going to have to reboot. – slm Jul 19 '18 at 13:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.