We are seeing a small cluster become unusable. Initially the same behavior was on a compute node, it is now occuring on the head node. I do not know if this is the underlying source, but for certain something is screwed up with the /tmp directory so even ls /tmp hangs and cannot be killed. (/tmp is under /, not nfs mounted, and I can see everything else, for instance /var/log, /proc etc.) Because so many daemons and running tasks expect to access /tmp, it makes sense to me this is a significant part of the problem.
A hard reboot solves the problem for a while, but that is not a long term cure.
Suggestions welcome, not simply to run "ls -ld /tmp &" which won't do more than ls ...
Note: when the problem occurs, /tmp is screwed up; otherwise (as now) it is fine:
[ldm@head ~]$ df -h /tmp
Filesystem Size Used Avail Use% Mounted on
/dev/md126 221G 143G 78G 65% /
[ldm@head ~]$ ls -ld /tmp
drwxrwxrwt. 12 root root 20480 Jan 26 08:45 /tmp
For reference:
uname -a
"Linux head.cluster 3.10.0-1062.1.1.el7.x86_64 #1 SMP Fri Sep 13 22:55:44 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
cat /etc/os-release
NAME="CentOS Linux"
VERSION="7 (Core)"
The problem is intermittent. It just reappeared on one of the compute nodes, with dmseg -H showing at the end:
[Feb 7 00:51] INFO: task kworker/4:2:20770 blocked for more than 120 seconds.
[ +0.007162] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[ +0.008112] kworker/4:2 D ffff985b47709040 0 20770 2
[ +0.007307] Workqueue: events xprt_rdma_connect_worker [rpcrdma]
[ +0.006210] Call Trace:
[ +0.002638] [] schedule+0x29/0x70
[ +0.005159] [] schedule_timeout+0x221/0x2d0
[ +0.006035] [] ? mthca_modify_qp+0x8f/0x310 [ib_mthca]
[ +0.006988] [] wait_for_completion+0xfd/0x140
[ +0.006204] [] ? wake_up_state+0x20/0x20
[ +0.005776] [] __ib_drain_sq+0x181/0x1c0 [ib_core]
[ +0.006638] [] ? ib_sg_to_pages+0x1a0/0x1a0 [ib_core]
[ +0.006902] [] ib_drain_sq+0x25/0x30 [ib_core]
[ +0.006292] [] ib_drain_qp+0x12/0x30 [ib_core]
[ +0.006291] [] rpcrdma_ep_disconnect+0x58/0x150 [rpcrdma]
[ +0.007244] [] rpcrdma_ep_connect+0x139/0x400 [rpcrdma]
[ +0.007073] [] ? wake_up_atomic_t+0x30/0x30
[ +0.006022] [] xprt_rdma_connect_worker+0x33/0x60 [rpcrdma]
[ +0.007505] [] process_one_work+0x17f/0x440
[ +0.006022] [] worker_thread+0x126/0x3c0
[ +0.005765] [] ? manage_workers.isra.25+0x2a0/0x2a0
[ +0.006725] [] kthread+0xd1/0xe0
[ +0.005071] [] ? insert_kthread_work+0x40/0x40
[ +0.006285] [] ret_from_fork_nospec_begin+0x21/0x21
[ +0.006714] [] ? insert_kthread_work+0x40/0x40
ls -ld /tmp
drwxrwxrwt 8 root root 169 Feb 7 11:28 /tmp
ls -ld /boot
dr-xr-xr-x 5 root root 4096 Jan 16 12:09 /boot
ls -ld / hangs -- it seems that NFS mount(s) have died.
ls -ld /tmp &to list the type of /tmp in the background. Having it in the background means even it it hangs you will still have your shell session. Second step would be to examine the mounts.ls -ld /tmp &as directed; The/etc/os-releaseanduname -aare kind of irrelevant; it's not like there are a thousand different variants of Centos 7.dmesg -Hand see if the listing produced includes any error messages suggesting filesystem corruption or hardware issues. On CentOS 7, hanging processes may typically also produce messages indmesgoutput once they've hung for more than two minutes or so, but those messages are not very interesting since you already know thatlsprocesses are hanging.