Mysterious bash instances using a lot of CPU, how can I debug?

Question

I seem to have multiple bash processes running that are taking up most of my CPU. This is the output of top -c:

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND         
20116 terdon    20   0 35288  14m  292 R 400.0  0.2   0:00.43 /bin/bash       
20106 terdon    20   0 35992  15m  280 R  95.9  0.2   0:00.65 /bin/bash       
20105 terdon    20   0     0    0    0 R  57.6  0.0   0:00.83 [bash]          

This is the output of ps aux | grep bash | head -3:

terdon  7487 45.3  0.0      0     0 ?        R    19:31   0:01 [bash]
terdon  7488 66.0  0.0      0     0 ?        R    19:31   0:01 [bash]
terdon  7530 23.0  0.2  37984 17408 ?        R    19:31   0:00 /bin/bash

The PIDs change every time I run the command so it looks like something is constantly respawning bash.

Details:

• There are multiple [bash] entries. If I understand correctly [process name] means that the process was launched with no command line arguments.
• The PIDs change so something is spawning these.
• I have logged out and logged back in (I am working in Cinnamon) and the problem persists.

Now, I imagine this will go away if I restart, my main question is what can I use to track these processes down?

top -c does not help, pgrep bash just gives me different lists of PIDs, lsof /bin/bash just lists running bash instances and pstree shows them as independent processes.

In case it is relevant, I am running Linux Mint Debian, kernel 3.2.0-4-amd64, GNU bash, version 4.2.36(1)-release.

---

EDIT:

I have since rebooted (I had to) and, as expected, the problem has gone away. I am still interested in useful suggestions of how to track down such processes though.

Answer 1

Take a look at the output of lsof | grep 'bash.*cwd'. That will tell you the current working directories of the processes.

If you have pstree, take a look at its output. If not, take a look at the output of ps aux -H. That will tell you which processes own these mystery processes.

Start looking through configuration files for anything suspicious. Here's an incomplete list of ones you should check:

~/.bash*
~/.profile
/etc/profile
/etc/bash*
/etc/cron.*/*

The [process name] means that ps can't find that process' arguments, including argument 0 which contains the name of the file that was executed to create the process. That means lsof /bin/bash won't find these processes.

Answer 2

You can also take a look at output of:

ps -p <pid> -o pid,user,tty,comm,args

Some shell scripts will appear as /bin/bash or perl in the output of top but args argument in the output of ps should give you the exact script running.