Is there some way I can check which of my processes the kernel has killed? Sometimes I log onto my server and find that something that should've run all night just stopped 8 hours in and I'm unsure if it's the applications doing or the kernels.
If the kernel killed a process (because the system ran out of memory), there will be a kernel log message. Check in
Note that if the OOM-killer (out-of-memory killer) triggered, it means you don't have enough virtual memory. Add more swap (or perhaps more RAM).
Some process crashes are recorded in kernel logs as well (e.g. segmentation faults).
If the processes were started from cron, you should have a mail with error messages. If the processes were started from a shell in a terminal, check the errors in that terminal. Run the process in
Process Accounting could help here.
Then try commands like:
or on Ubuntu:
So as far as I can see, you'd have to write your own C program to access the information.
Here's a version that prints the exit code.
The last two fields are "S" for signaled and "E" for exited, followed by the signal number or exit status.
So in your case, you're probably looking for "S 15" meaning it got a SIGTERM.
Compared to "E 0" which means the process exited without an error.
Only minimally tested.
This command will tell you the what are the services are currently running and which are not started or stopped..