We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. For example I have seen that a lot of CPUs on one of the nodes are assigned to me despite the fact that I had killed the allocated resources with scancel a couple of days ago! I want to find them and kill them.

On my local machine I have generated public and private ssh keys with ssh-keygen so now I can login to any of those machines with ssh foo, ssh [email protected], ssh [email protected] ... but the names of those nodes are not in an order. If one login to one of those nodes and run sinfo this is the result:

debug          up   infinite      3    mix baz[080-081,083]
debug          up   infinite      2  alloc grault,baz082
debug          up   infinite     13   idle baz[061-070],corge,bar,quux
gpu_p100       up   infinite      1    mix baz080
gpu_titan-x    up   infinite      2    mix baz[081,083]
gpu_titan-x    up   infinite      1  alloc baz082
r730           up   infinite      1    mix baz080
t630           up   infinite      2    mix baz[081,083]
t630           up   infinite      1  alloc baz082
r930           up   infinite      1  alloc grault
m610           up   infinite     10   idle baz[061-070]
r720           up   infinite      1   idle corge
r815           up   infinite      1   idle bar
sm1u           up   infinite      1   idle quux
main*          up   infinite      3    mix baz[080-081,083]
main*          up   infinite      2  alloc grault,baz082
main*          up   infinite     12   idle baz[061-070],bar,quux

where baz[081-083] is referring to 3 nodes baz081, baz082 and baz083. no if I ssh into any of these nodes I can list all the processes assigned to a specific user by:

ps -A | grep user1

but that would take a lot of time. How can I automate this process:

  1. login to one of the nodes
  2. run sinfo
  3. extract the information and make a list of strings from the last column of sinfo command
  4. find all the running processes with a specific user user1 and prints to the terminal

How can I write an script, preferably compatible with Cmder/ConEmu, to automate these steps.

2 Answers 2


This will try to login to all nodes and print processes of user1:

    for hostList in $(sinfo -h | awk '{print $6}' | sort -u); do
            for host in $(scontrol show hostname $hostList); do
                    echo $host; ssh $host "ps aux | grep user1";

However, as you probably do not want to login to offline nodes, extend the sinfo command to:

        for hostList in $(sinfo -h | grep -v down | awk '{print $6}' | sort -u); do

There might be even better ways to accomplish this with better knowledge of sinfo / scontrol but this script should do the job.

However, if this happens a lot you might consider, talking to your administrators as orphaned processes are bad for the cluster performance in general. They should ideally be cleaned by the system automatically on a regular basis.

  • thanks a lot. I'm trying to go through the code step by step. The first issue is that to get the sinfo you need to first ssh into one of the servers, lets say foo
    – Foad
    Sep 5, 2018 at 11:46
  • 1
    You can put the code in a file "checkAllServers.sh" in your home on the server and run it from your machine via ssh foo bash checkAllServers.h or you can run the whole code in one line via: ssh foo 'for hostList in $(sinfo -h | awk '{print $6}' | sort -u); do for host in $(scontrol show hostname $hostList); do echo $host; ssh $host "ps aux | grep user1"; done; done;' Sep 6, 2018 at 7:38
  • If i want to run the bash on one of the nodes then I have to put my private key there, which can be read by everyone else. What I want is to run the script from my own computer(s). Is there any way to pip the output of a command ran on the server to the local machine?
    – Foad
    Sep 6, 2018 at 9:12
  • Ok, I think if we replace sinfo -h with ssh foo "sinfo -h;" the problem is mostly solve the only issue is that the -u flag for sort doesn't work and leads to the The system cannot find the file specified. error.
    – Foad
    Sep 6, 2018 at 10:45

Based on Noco's answer and further investigation I wrote the perfect script for my own purpose:

hostList=$(ssh foo "sinfo -h" | awk '{print $6}' | sed 's/,/\n/g' | awk '!seen[$0]++' | sed '/\[/d')
for host in $hostList; do
    echo $host;
    ssh "$1@$host.uni.com" "ps aux | grep $2";
    #ssh "$1@$host.uni.com" "pkill -f $2";

where foo is the main node in my case. This script accept two arguments as sh script arg1 arg2 where arg1 is the user1 as indicated in OP and arg2 is the keyword it will search for. If one wants to remove all the processes assigned to that key word , he/she should uncomment the line with pkill command.

What I observed is that the nodes showing as baz[080-081,083] are all the same in may case so need to search all.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .