I am trying to find a root cause for server hung.
I found that a process crashed with process id 14900 and the following are the log in message. No core dump is saved as it is not related to any package (ProcessUnpackaged=no).
May 25 15:31:41 myserver abrt: Saved core dump of pid 14900 (/NFS_share/work_dir/freac/FREAC.Linux-2.6-x86_64-Release) to /var/spool/abrt/ccpp-2016-05-25-15:31:41-14900 (11644928 bytes) May 25 15:31:52 myserver abrtd: Sending an email... May 25 15:31:52 myserver abrtd: Email was sent to: root@localhost May 25 15:31:52 myserver abrtd: Duplicate: UUID May 25 15:31:52 myserver abrtd: DUP_OF_DIR: /var/spool/abrt/ccpp-2016-05-17-10:25:46-48111 May 25 15:31:52 myserver abrtd: Problem directory is a duplicate of /var/spool/abrt/ccpp-2016-05-17-10:25:46-48111 May 25 15:31:52 myserver abrtd: Deleting problem directory ccpp-2016-05-25-15:31:06-12824 (dup of ccpp-2016-05-17-10:25:46-48111) May 25 15:31:52 myserver abrtd: Failed to open connection to "system" message bus: Failed to connect to socket /var/run/dbus/system_bus_socket: Connection refused May 25 15:31:52 myserver abrtd: Directory 'ccpp-2016-05-25-15:31:41-14900' creation detected May 25 15:31:52 myserver abrtd: Executable '/NFS_share/work_dir/freac/FREAC.Linux-2.6-x86_64-Release' doesn't belong to any package May 25 15:31:52 myserver abrtd: 'post-create' on '/var/spool/abrt/ccpp-2016-05-25-15:31:41-14900' exited with 1 May 25 15:31:52 myserver abrtd: Corrupted or bad directory /var/spool/abrt/ccpp-2016-05-25-15:31:41-14900, deleting
There is another process 14939 which is possibly child process of 14900 hung and this caused increase in load and eventually hung of the server.
May 25 15:33:44 myserver ntpd: synchronized to 10.171.8.5, stratum 3 May 25 15:35:10 myserver kernel: INFO: task FREAC.Linux-2.6:14939 blocked for more than 120 seconds. May 25 15:35:10 myserver kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. May 25 15:35:10 myserver kernel: FREAC.Linux-2 D 00000000ffffffff 0 14939 14658 0x10000084 May 25 15:35:10 myserver kernel: ffff8835d4ebd988 0000000000000046 ffff8835d4ebd908 ffffffffa0844e00 May 25 15:35:10 myserver kernel: ffff8828a4b61440 ffff881fedd4a540 ffff8835d4000001 ffffffff81129607 May 25 15:35:10 myserver kernel: ffff883f4c39baf8 ffff8835d4ebdfd8 000000000000fb88 ffff883f4c39baf8 May 25 15:35:10 myserver kernel: Call Trace:
There was issue with dbus that we had not fixed at that time, but could it be the reason for the failure of the child process 14939. I don't know exactly what's the purpose of dbus.
I couldn't get any details regarding the process as the server went to hung due to increase in load and we had to restart it. However, we fixed the dbus issue after restart.
Some recent understanding after a brief look at this link: https://dbus.freedesktop.org/doc/dbus-tutorial.html
dbus is required for interprocess communication (IPC) (means communication with another process to send messages and nothing to do with parent or child calls).
There is a statement:
The systemwide and per-user daemons are separate. Normal within-session IPC does not involve the systemwide message bus process and vice versa.
So, what does vice versa mean here - Does IPC not with-in-session require dbus process (systemwide or user)?
If this is right, then communication between 14939 and 14900 doesn't require dbus at all as they are with-in-session? Or may be not, may be init has inherited one or both the processes and hence, dbus is required.
Then another question bugs me - actually the dbus issue started after a recent restart of that server and after some few days the server went to hung. If dbus is required for all these processes to run successfully, why weren't there any processes hung in those few days after restart.
Kindly just try to answer the actual question regarding dbus, if the rest of the question is too broad.
And also this: Why do I need dbus? does clarify few things regarding dbus.