How to optimize 'grep'-and-save?

Question

I want to know on which node my calculations were running on our cluster in case the node crashes and I get no files back to know on which node I have to search.

For that case I wrote a little script that gets the job done - except for some cases that I somehow do not get to manage.

I want to parse from the following output the JobID, the Queue, the Jobname and the node on which it is running - if it is running.

my12name@omega:/some/fancy/path> qstat -n -u my12name

omega.cluster: 
                                                                   Req'd  Req'd   Elap
Job ID               Username Queue    Jobname    SessID NDS   TSK Memory Time  S Time
-------------------- -------- -------- ---------- ------ ----- --- ------ ----- - -----
2974949.omega.cluste my12name  short    j-M0044_td  21582     1   8 12288m 500:0 R 120:1
   node54/7+node54/6+node54/5+node54/4+node54/3+node54/2+node54/1+node54/0
2974950.omega.cluste my12name  short    j-M0045_td    --      1   8 12288m 500:0 R 120:2
   octo08/7+octo08/6+octo08/5+octo08/4+octo08/3+octo08/2+octo08/1+octo08/0
2974951.omega.cluste my12name  short    j-M0046_td    --      1   8 12288m 500:0 R 120:3
   il41/7+il41/6+il41/5+il41/4+il41/3+il41/2+il41/1+il41/0
2974951.omega.cluste my12name  short    j-M0046_td    --      1   8 12288m 500:0 R 120:3
   il41/15+il41/14+il41/13+il41/12+il41/11+il41/10+il41/9+il41/8
   +il41/7+il41/6+il41/5+il41/4+il41/3+il41/2+il41/1+il41/0
2976371.omega.cluste my12name  short    j-M0049_fr    --      1   8 12288m 500:0 Q   -- 
    -- 

My script for that currently looks like this:

#!/bin/bash                                                                                                                                                                                                                           

qstat -n -u my12name |grep -v "[ ]+" > DeleteMeQuick1
cat DeleteMeQuick1|grep 'node\|octo\|il' |tr "/" " "|awk '{print $1}' > DeleteMeQuick2
cat DeleteMeQuick1|grep 'my12name'|awk '{print $1, $3, $4}' > DeleteMeQuick3
awk 'NR==FNR{a[NR]=$0; next} {print a[FNR], $0}' DeleteMeQuick2 DeleteMeQuick3 >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$2]++' > DeleteMeQuick4
cat DeleteMeQuick4 > ~/.qstat_history
rm DeleteMeQuick*

It takes the query and

1. searches for lines that don't start with a plus-sign and saves them to a temporary file
2. from this file it searches for lines beginning with the possible node-names and saves them to a second temporary file
3. it also takes from the non-node-lines the JobID, etc.
4. it adds from each temporary file the 1st, 2nd, 3rd, ... line side by side to my history file
5. it deletes duplicate entries from the history file
6. it deletes temporary files

The ouput in my history file then looks like this:

...
octo11 2955937.omega.cluste big16 j-M0044_op
node55 2956189.omega.cluste short j-M0045_op
il11 2963103.omega.cluste oshort n2.sh
....

---

Example for my first grep

Normal output:

2976388.omega.cluste my12name  big24    n2.sh       28095     1  --   48gb 300:0 R   -- 
   il32/23+il32/22+il32/21+il32/20+il32/19+il32/18+il32/17+il32/16+il32/15
   +il32/14+il32/13+il32/12+il32/11+il32/10+il32/9+il32/8+il32/7+il32/6+il32/5
   +il32/4+il32/3+il32/2+il32/1+il32/0

And when using ...|grep -v '[ ]+':

2976388.omega.cluste my12name  big24    n2.sh       28095     1  --   48gb 300:0 R   -- 
   il32/23+il32/22+il32/21+il32/20+il32/19+il32/18+il32/17+il32/16+il32/15

---

What could be a better yet more efficient way to approach this task?

Currently my script is not able to ignore jobs that haven't started (the job line contains the "Q" and the node line contains only the double hyphens).

Answer 1

Thak you, cas. Your $12 idea helped me to get this new version.

I was able to reduce it to the following:

#!/bin/bash
qstat -n -u my12name|grep -v '[ ]---\|[ ]+\|Username\|Elap'|paste - -|sed 's/\/.*//g'|grep -v ' Q ' >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$1]++'|sed '/^\s*$/d' > qstat_history.tmp
mv qstat_history.tmp ~/.qstat_history

It's deleting the beginning lines (Username, Elap, ---) and the lines that start with a plus sign, then puts the node line behind the job-line (paste - -), deletes the node-stuff behind the slash (s/\/.*//g) and greps every job that is running resp. does not grep jobs that are waiting (|grep -v ' Q '). Whatever is left gets added to my .qstat_history file.

Next is to delete duplicate lines with awk, save it to temporary file and move the temporary file to be the new .qstat_history.

---

I was shown a function of qstat that is very hidden (-1), that puts the node-line directly behind the job-line. This easifies everything a bit more to give the following version:

#!/bin/bash
qstat -u my12name -n -1|sed 's/\/.*//g'|grep ' R ' >> ~/.qstat_history
cat ~/.qstat_history|awk '!NF ||!seen[$1]++' > qstat_history.tmp
mv qstat_history.tmp ~/.qstat_history