1

We have a cluster in our university controlled by SLURM. The issue is that I have observed sometimes some of the resources are not being realised even though they do not show up in the squeue command. For example I have seen that a lot of CPUs on one of the nodes are assigned to me despite the fact that I had killed the allocated resources with scancel a couple of days ago! I want to find them and kill them.

On my local machine I have generated public and private ssh keys with ssh-keygen so now I can login to any of those machines with ssh foo, ssh user1@bar.uni.com, ssh user1@baz.uni.com ... but the names of those nodes are not in an order. If one login to one of those nodes and run sinfo this is the result:

PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
debug          up   infinite      3    mix baz[080-081,083]
debug          up   infinite      2  alloc grault,baz082
debug          up   infinite     13   idle baz[061-070],corge,bar,quux
gpu_p100       up   infinite      1    mix baz080
gpu_titan-x    up   infinite      2    mix baz[081,083]
gpu_titan-x    up   infinite      1  alloc baz082
r730           up   infinite      1    mix baz080
t630           up   infinite      2    mix baz[081,083]
t630           up   infinite      1  alloc baz082
r930           up   infinite      1  alloc grault
m610           up   infinite     10   idle baz[061-070]
r720           up   infinite      1   idle corge
r815           up   infinite      1   idle bar
sm1u           up   infinite      1   idle quux
main*          up   infinite      3    mix baz[080-081,083]
main*          up   infinite      2  alloc grault,baz082
main*          up   infinite     12   idle baz[061-070],bar,quux

where baz[081-083] is referring to 3 nodes baz081, baz082 and baz083. no if I ssh into any of these nodes I can list all the processes assigned to a specific user by:

ps -A | grep user1

but that would take a lot of time. How can I automate this process:

  1. login to one of the nodes
  2. run sinfo
  3. extract the information and make a list of strings from the last column of sinfo command
  4. find all the running processes with a specific user user1 and prints to the terminal

How can I write an script, preferably compatible with Cmder/ConEmu, to automate these steps.

2

This will try to login to all nodes and print processes of user1:

    for hostList in $(sinfo -h | awk '{print $6}' | sort -u); do
            for host in $(scontrol show hostname $hostList); do
                    echo $host; ssh $host "ps aux | grep user1";
            done;
    done;

However, as you probably do not want to login to offline nodes, extend the sinfo command to:

        for hostList in $(sinfo -h | grep -v down | awk '{print $6}' | sort -u); do

There might be even better ways to accomplish this with better knowledge of sinfo / scontrol but this script should do the job.

However, if this happens a lot you might consider, talking to your administrators as orphaned processes are bad for the cluster performance in general. They should ideally be cleaned by the system automatically on a regular basis.

  • thanks a lot. I'm trying to go through the code step by step. The first issue is that to get the sinfo you need to first ssh into one of the servers, lets say foo – Foad Sep 5 '18 at 11:46
  • 1
    You can put the code in a file "checkAllServers.sh" in your home on the server and run it from your machine via ssh foo bash checkAllServers.h or you can run the whole code in one line via: ssh foo 'for hostList in $(sinfo -h | awk '{print $6}' | sort -u); do for host in $(scontrol show hostname $hostList); do echo $host; ssh $host "ps aux | grep user1"; done; done;' – Nico Mittenzwey Sep 6 '18 at 7:38
  • If i want to run the bash on one of the nodes then I have to put my private key there, which can be read by everyone else. What I want is to run the script from my own computer(s). Is there any way to pip the output of a command ran on the server to the local machine? – Foad Sep 6 '18 at 9:12
  • Ok, I think if we replace sinfo -h with ssh foo "sinfo -h;" the problem is mostly solve the only issue is that the -u flag for sort doesn't work and leads to the The system cannot find the file specified. error. – Foad Sep 6 '18 at 10:45
1

Based on Noco's answer and further investigation I wrote the perfect script for my own purpose:

#!/bin/bash 
hostList=$(ssh foo "sinfo -h" | awk '{print $6}' | sed 's/,/\n/g' | awk '!seen[$0]++' | sed '/\[/d')
for host in $hostList; do
    echo $host;
    ssh "$1@$host.uni.com" "ps aux | grep $2";
    #ssh "$1@$host.uni.com" "pkill -f $2";
done;

where foo is the main node in my case. This script accept two arguments as sh script arg1 arg2 where arg1 is the user1 as indicated in OP and arg2 is the keyword it will search for. If one wants to remove all the processes assigned to that key word , he/she should uncomment the line with pkill command.

What I observed is that the nodes showing as baz[080-081,083] are all the same in may case so need to search all.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.