2

I want to know on which node my calculations were running on our cluster in case the node crashes and I get no files back to know on which node I have to search.

For that case I wrote a little script that gets the job done - except for some cases that I somehow do not get to manage.

I want to parse from the following output the JobID, the Queue, the Jobname and the node on which it is running - if it is running.

my12name@omega:/some/fancy/path> qstat -n -u my12name

omega.cluster: 
                                                                   Req'd  Req'd   Elap
Job ID               Username Queue    Jobname    SessID NDS   TSK Memory Time  S Time
-------------------- -------- -------- ---------- ------ ----- --- ------ ----- - -----
2974949.omega.cluste my12name  short    j-M0044_td  21582     1   8 12288m 500:0 R 120:1
   node54/7+node54/6+node54/5+node54/4+node54/3+node54/2+node54/1+node54/0
2974950.omega.cluste my12name  short    j-M0045_td    --      1   8 12288m 500:0 R 120:2
   octo08/7+octo08/6+octo08/5+octo08/4+octo08/3+octo08/2+octo08/1+octo08/0
2974951.omega.cluste my12name  short    j-M0046_td    --      1   8 12288m 500:0 R 120:3
   il41/7+il41/6+il41/5+il41/4+il41/3+il41/2+il41/1+il41/0
2974951.omega.cluste my12name  short    j-M0046_td    --      1   8 12288m 500:0 R 120:3
   il41/15+il41/14+il41/13+il41/12+il41/11+il41/10+il41/9+il41/8
   +il41/7+il41/6+il41/5+il41/4+il41/3+il41/2+il41/1+il41/0
2976371.omega.cluste my12name  short    j-M0049_fr    --      1   8 12288m 500:0 Q   -- 
    -- 

My script for that currently looks like this:

#!/bin/bash                                                                                                                                                                                                                           

qstat -n -u my12name |grep -v "[ ]+" > DeleteMeQuick1
cat DeleteMeQuick1|grep 'node\|octo\|il' |tr "/" " "|awk '{print $1}' > DeleteMeQuick2
cat DeleteMeQuick1|grep 'my12name'|awk '{print $1, $3, $4}' > DeleteMeQuick3
awk 'NR==FNR{a[NR]=$0; next} {print a[FNR], $0}' DeleteMeQuick2 DeleteMeQuick3 >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$2]++' > DeleteMeQuick4
cat DeleteMeQuick4 > ~/.qstat_history
rm DeleteMeQuick*

It takes the query and

  1. searches for lines that don't start with a plus-sign and saves them to a temporary file
  2. from this file it searches for lines beginning with the possible node-names and saves them to a second temporary file
  3. it also takes from the non-node-lines the JobID, etc.
  4. it adds from each temporary file the 1st, 2nd, 3rd, ... line side by side to my history file
  5. it deletes duplicate entries from the history file
  6. it deletes temporary files

The ouput in my history file then looks like this:

...
octo11 2955937.omega.cluste big16 j-M0044_op
node55 2956189.omega.cluste short j-M0045_op
il11 2963103.omega.cluste oshort n2.sh
....

Example for my first grep

Normal output:

2976388.omega.cluste my12name  big24    n2.sh       28095     1  --   48gb 300:0 R   -- 
   il32/23+il32/22+il32/21+il32/20+il32/19+il32/18+il32/17+il32/16+il32/15
   +il32/14+il32/13+il32/12+il32/11+il32/10+il32/9+il32/8+il32/7+il32/6+il32/5
   +il32/4+il32/3+il32/2+il32/1+il32/0

And when using ...|grep -v '[ ]+':

2976388.omega.cluste my12name  big24    n2.sh       28095     1  --   48gb 300:0 R   -- 
   il32/23+il32/22+il32/21+il32/20+il32/19+il32/18+il32/17+il32/16+il32/15

What could be a better yet more efficient way to approach this task?

Currently my script is not able to ignore jobs that haven't started (the job line contains the "Q" and the node line contains only the double hyphens).

  • This question is confusing. The first line filters out anything beginning with a space (not a plus), so the line that searches for node|octo|il should output nothing. The third line searches for me12name, but the data contains my12name. Please clarify these things by editing the question. – Tom Zych Mar 6 '16 at 15:00
  • I don't get the first problem, as it works ... maybe my description might be wrong, but it does what it is supposed to do. I fixed the second problem which came from inconsitent "anonymization". – pH13 - Yet another Philipp Mar 6 '16 at 15:55
  • It looks like you're not posting the input and output lines as they are, but with extra newlines inserted at some point for readability. – Tom Zych Mar 6 '16 at 16:34
  • @TomZych: I appreciate that you want to help me. Maybe we could elaborate this in chat? I not sure if the comment section is the right place to work on that. – pH13 - Yet another Philipp Mar 6 '16 at 17:04
  • 1
    have you tried something simpler like qstat -n -u my12name | awk '/my12name/ {gsub(/\/.*/,"",$12) ; print $12, $1, $3, $4}' ? does that come close to the output you want? – cas Mar 7 '16 at 3:24
2

Thak you, cas. Your $12 idea helped me to get this new version.

I was able to reduce it to the following:

#!/bin/bash
qstat -n -u my12name|grep -v '[ ]---\|[ ]+\|Username\|Elap'|paste - -|sed 's/\/.*//g'|grep -v ' Q ' >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$1]++'|sed '/^\s*$/d' > qstat_history.tmp
mv qstat_history.tmp ~/.qstat_history

It's deleting the beginning lines (Username, Elap, ---) and the lines that start with a plus sign, then puts the node line behind the job-line (paste - -), deletes the node-stuff behind the slash (s/\/.*//g) and greps every job that is running resp. does not grep jobs that are waiting (|grep -v ' Q '). Whatever is left gets added to my .qstat_history file.

Next is to delete duplicate lines with awk, save it to temporary file and move the temporary file to be the new .qstat_history.


I was shown a function of qstat that is very hidden (-1), that puts the node-line directly behind the job-line. This easifies everything a bit more to give the following version:

#!/bin/bash
qstat -u my12name -n -1|sed 's/\/.*//g'|grep ' R ' >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$1]++' > qstat_history.tmp
mv qstat_history.tmp ~/.qstat_history
  • 1
    glad to have helped by pointing you in a simpler direction. BTW, you can get rid of the grep by using sed -n 's/\/.*//g ; / R / p' – cas Mar 7 '16 at 23:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.