1

I am trying to find a root cause for server hung.

I found that a process crashed with process id 14900 and the following are the log in message. No core dump is saved as it is not related to any package (ProcessUnpackaged=no).

May 25 15:31:41 myserver abrt[15298]: Saved core dump of pid 14900 (/NFS_share/work_dir/freac/FREAC.Linux-2.6-x86_64-Release) to /var/spool/abrt/ccpp-2016-05-25-15:31:41-14900 (11644928 bytes)
May 25 15:31:52 myserver abrtd: Sending an email...
May 25 15:31:52 myserver abrtd: Email was sent to: root@localhost
May 25 15:31:52 myserver abrtd: Duplicate: UUID
May 25 15:31:52 myserver abrtd: DUP_OF_DIR: /var/spool/abrt/ccpp-2016-05-17-10:25:46-48111
May 25 15:31:52 myserver abrtd: Problem directory is a duplicate of /var/spool/abrt/ccpp-2016-05-17-10:25:46-48111
May 25 15:31:52 myserver abrtd: Deleting problem directory ccpp-2016-05-25-15:31:06-12824 (dup of ccpp-2016-05-17-10:25:46-48111)
May 25 15:31:52 myserver abrtd: Failed to open connection to "system" message bus: Failed to connect to socket /var/run/dbus/system_bus_socket: Connection refused
May 25 15:31:52 myserver abrtd: Directory 'ccpp-2016-05-25-15:31:41-14900' creation detected
May 25 15:31:52 myserver abrtd: Executable '/NFS_share/work_dir/freac/FREAC.Linux-2.6-x86_64-Release' doesn't belong to any package
May 25 15:31:52 myserver abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-05-25-15:31:41-14900' exited with 1
May 25 15:31:52 myserver abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2016-05-25-15:31:41-14900, deleting

There is another process 14939 which is possibly child process of 14900 hung and this caused increase in load and eventually hung of the server.

May 25 15:33:44 myserver ntpd[4430]: synchronized to 10.171.8.5, stratum 3
May 25 15:35:10 myserver kernel: INFO: task FREAC.Linux-2.6:14939 blocked for more than 120 seconds.
May 25 15:35:10 myserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
May 25 15:35:10 myserver kernel: FREAC.Linux-2 D 00000000ffffffff     0 14939  14658 0x10000084
May 25 15:35:10 myserver kernel: ffff8835d4ebd988 0000000000000046 ffff8835d4ebd908 ffffffffa0844e00
May 25 15:35:10 myserver kernel: ffff8828a4b61440 ffff881fedd4a540 ffff8835d4000001 ffffffff81129607
May 25 15:35:10 myserver kernel: ffff883f4c39baf8 ffff8835d4ebdfd8 000000000000fb88 ffff883f4c39baf8
May 25 15:35:10 myserver kernel: Call Trace:

There was issue with dbus that we had not fixed at that time, but could it be the reason for the failure of the child process 14939. I don't know exactly what's the purpose of dbus.

I couldn't get any details regarding the process as the server went to hung due to increase in load and we had to restart it. However, we fixed the dbus issue after restart.

EDIT1:

Some recent understanding after a brief look at this link: https://dbus.freedesktop.org/doc/dbus-tutorial.html

dbus is required for interprocess communication (IPC) (means communication with another process to send messages and nothing to do with parent or child calls).

There is a statement:

The systemwide and per-user daemons are separate. Normal within-session IPC does not involve the systemwide message bus process and vice versa.

So, what does vice versa mean here - Does IPC not with-in-session require dbus process (systemwide or user)?

If this is right, then communication between 14939 and 14900 doesn't require dbus at all as they are with-in-session? Or may be not, may be init has inherited one or both the processes and hence, dbus is required.

Then another question bugs me - actually the dbus issue started after a recent restart of that server and after some few days the server went to hung. If dbus is required for all these processes to run successfully, why weren't there any processes hung in those few days after restart.

Kindly just try to answer the actual question regarding dbus, if the rest of the question is too broad.

Thank you!

EDIT 2:

And also this: Why do I need dbus? does clarify few things regarding dbus.

  • if you dont get answers, its because we dont want to start a flame war. in short, dbus is needed because systemd. And that exists primarily because RedHat. – Otheus Jun 6 '16 at 15:25
  • But you have 3 problems. First, the process crashing. Why is it crashing? Abrtd could help you figire it out but it cannot write its crash reports. So it then tries to connect to Dbus (why, im not sure) and fails there too. Check email as root... you will see the reports there. Maybe you will get insight into forst problem. – Otheus Jun 6 '16 at 15:26
  • @Otheus Okay, I will check the mails. I usually never check mails. Let's see. thanks! – GP92 Jun 6 '16 at 17:46
  • @Otheus Why it tries to connect to dbus. I am assuming that it's trying to send message to another process 14939 related to it - wild guess! – GP92 Jun 6 '16 at 17:48
  • @Otheus The message to root doesn't contain the details of the failed process, but about completely different process run by a different user. So, no help. – GP92 Jun 7 '16 at 7:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.