17

I have a process which several times now has stopped responding and appears to be completely locking up. It doesn't respond to any attempt at strace or peeking with gdb (gdb just hangs on a wait4() syscall). The process is runnable, and is not waiting on a syscall (/proc/X/syscall: running) or in uninterruptable sleep (/proc/X/status: State: R (running)).

What state is this process in exactly? Is this possibly a kernel bug of some type?

The process is redis, and this has happened a few times now. Only thing that can kill the process is a reboot, it seems. OS is Cent 7.

Edit: Kernel version is 3.10.0-123.13.2.el7.x86_64. Trying an update to 3.10.0-229.11.1.el7 to see if that makes any difference.

  • What version of GDB is it using? According to stackoverflow.com/questions/8978777/… a newer version might work better. – Greg Bray Aug 4 '15 at 1:18
  • Currently it looks like the investigation is more kernel side because of the special way it hangs, but if you don't mind, could you add a few Redis specific informations? What the process is doing while it blocks and things like that. I got a few info from Nick Craver via Twitter, apparently Redis is loading a large data set when this happens, is the data set loaded just restarting the process or in some other way (for example via DEBUG RELOAD, or pipelining large amounts of data)? Thanks. – antirez Aug 5 '15 at 6:27
  • @antirez The data set is being loaded by an rdb copy from another redis instance. The lockups occur after redis starts up and reads in the giant rdb. Notably it doesn't always lockups during this, just sometimes. – alienth Aug 6 '15 at 0:00
  • 1
    I only had this sort of issues when having IO errors. Could you please tell us about the dmesg output? – Ho1 Aug 7 '15 at 21:14
  • 3
    What does /proc/<pid>/stack (and /proc/<pid>/task/*/stack) contain? Has that process got several threads? – Stéphane Chazelas Aug 9 '15 at 20:49
2
+100

wait4 is a syscall indicating the process is waiting for one of his child termination. This may points some issue with the signal handling.

A bit brutal, but you may try to kill the hierarchy of the app : kill -15 -$YourRedisPID. The - before the PID means "the PID and its children". As it seems to be waiting for a child termination, it may unlock it.

If it's not working, let's check deeper : find your signal process status with grep ^Sig /proc/$YourRedisPID/status

You'll see some stuff like :

SigQ:   8/62777
SigPnd: 0000000000000000
SigBlk: 0000000000000000
SigIgn: 0000000000000080
SigCgt: 0000000180004023

As defined in "fs/proc/array.c" of the kernel source, the "SigQ" is the number of signals pending / the limit of pending signals.

If the number of signal is too high, it may indicate your "SIGKILL" is not handled at all. I'm still checking the "kernel/signal.c" file to understand the signal management of these special signals.

For a direct understanding of the output, try this one-liner : awk 'BEGIN{print "ibase=16;obase=2;"} /^Sig...:/{ print toupper($2)}' /proc/$YourRedisPID/status | BC_LINE_LENGTH=0 bc

This outputs me :

0
0
10000000
110000000000000000100000000100011

Let's start by sending us this output. I'll update the post as required.

|improve this answer|||||
  • The process isn't in wait4(), gdb is hung on a wait4() when trying to access the process. The process itself isn't in any syscall. Also, the hung process has no children. Unfortunately I had to reboot the box. I'll gather the data you requested once the issue reoccurs. – alienth Aug 13 '15 at 17:15
  • Output here: gist.githubusercontent.com/alienth/23685ad2ea46a7eade56/raw/… Once again, proc is ignoring SIGKILL. It isn't in a syscall. Proc also ignores SIGTERM. – alienth Aug 24 '15 at 20:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.