Text processing - Building a slurm topology.conf file from ibnetdiscover output

Question

First things first: no knowledge of either slurm or Infiniband is required - this is a purely text processing problem.
Second - I'm aware of ib2slurm - the code is somehow broken and quite possibly outdated - it core dumps each time it runs regardless of the existence or format of a map file.

I can reduce the output of ibnetdiscover to 37 line blocks each of the form:

Switch  36 "S-0002c90200423e70"     # "MF0;ibsw20:SX6036/U1" enhanced port 0 lid 3 lmc 0
[1] "H-0002c903000c26f2"[1](2c903000c26f3)      # "compute061 HCA-1" lid 49 4xQDR
[2] "H-0002c903000bf36e"[1](2c903000bf36f)      # "compute060 HCA-1" lid 1 4xQDR
[3] "H-0002c903000bf35a"[1](2c903000bf35b)      # "compute063 HCA-1" lid 28 4xQDR
[4] "H-0002c903000c2646"[1](2c903000c2647)      # "compute062 HCA-1" lid 25 4xQDR
[5] "H-0002c903000bf35e"[1](2c903000bf35f)      # "compute064 HCA-1" lid 31 4xQDR
[6] "H-0002c903000c26de"[1](2c903000c26df)      # "compute065 HCA-1" lid 47 4xQDR
[7] "S-0002c90200423e80"[31]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[8] "S-0002c90200423e80"[32]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[9] "S-0002c90200423e80"[33]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[10]    "S-0002c90200423e80"[34]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[11]    "S-0002c90200423e80"[35]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[12]    "S-0002c90200423e80"[36]        # "Infiniscale-IV Mellanox Technologies" lid 6 4xQDR
[13]    "S-0002c90200423eb8"[35]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[14]    "S-0002c90200423eb8"[36]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[15]    "S-0002c90200423eb8"[33]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[16]    "S-0002c90200423eb8"[34]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[17]    "S-0002c90200423eb8"[31]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[18]    "S-0002c90200423eb8"[32]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[19]    "S-0002c90200423ee0"[31]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[20]    "S-0002c90200423ee0"[32]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[21]    "S-0002c90200423ee0"[33]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[22]    "S-0002c90200423ee0"[34]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[23]    "S-0002c90200423ee0"[35]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[24]    "S-0002c90200423ee0"[36]        # "Infiniscale-IV Mellanox Technologies" lid 15 4xQDR
[25]    "H-0002c903000c26fa"[1](2c903000c26fb)      # "compute046 HCA-1" lid 112 4xQDR
[26]    "H-0002c903000c26e2"[1](2c903000c26e3)      # "compute047 HCA-1" lid 63 4xQDR
[27]    "H-0002c903000c263a"[1](2c903000c263b)      # "compute048 HCA-1" lid 59 4xQDR
[28]    "H-0002c903000c27c2"[1](2c903000c27c3)      # "compute049 HCA-1" lid 117 4xQDR
[29]    "H-0002c903000c27a6"[1](2c903000c27a7)      # "compute051 HCA-1" lid 34 4xQDR
[30]    "H-0002c903000c2732"[1](2c903000c2733)      # "compute050 HCA-1" lid 22 4xQDR
[31]    "H-0002c903000c265e"[1](2c903000c265f)      # "compute052 HCA-1" lid 29 4xQDR
[32]    "H-0002c903000c266a"[1](2c903000c266b)      # "compute055 HCA-1" lid 32 4xQDR
[33]    "H-0002c903000c264e"[1](2c903000c264f)      # "compute054 HCA-1" lid 26 4xQDR
[34]    "H-0002c903000c26ee"[1](2c903000c26ef)      # "compute056 HCA-1" lid 48 4xQDR
[35]    "H-0002c903000bf246"[1](2c903000bf247)      # "compute057 HCA-1" lid 33 4xQDR
[36]    "H-0002c903000c27ca"[1](2c903000c27cb)      # "compute053 HCA-1" lid 44 4xQDR

and can extract the node name, e.g. compute061 using awk or sed.

I would like to get a single row for each block starting with switch name followed by node names, i.e: ibsw20 compute061 compute060 compute063 compute062 compute064 compute065 compute046 compute047 compute048 compute049 compute051 compute050 compute052 compute055 compute054 compute056 compute057 compute053

I plan to use slurm's scontrol show hostlist "<nodename> <nodename> ..." to compress several nodes into a single entity to push into slurm's topology.conf file which must have the form:

SwitchName=ibsw20 Nodes=compute[046-057,060-061]

Any ideas?

I should mention after all the switch mappings, the ibnetdiscover file continues with the reverse - a node-by-node mapping to switches, in the form:

vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903000bf371
caguid=0x2c903000bf36e
Ca  1 "H-0002c903000bf36e"      # "compute060 HCA-1"
[1](2c903000bf36f)  "S-0002c90200423e70"[2]     # lid 1 lmc 0 "MF0;ibsw20:SX6036/U1" lid 3 4xQDR

Each block separated by empty lines.

A reduced question that can get me started - how do I parse several lines of text into a single row, extracting different parts of each row (treating header and body rows differently) and discarding rows which do not contain relevant data?

EDIT: The blocks might not be full - if nothing is connected to some of the ports in some of the switches, then the output will skip those line, and can result in something like:

Switch  36 "S-0002c90200423e70"     # "MF0;ibsw20:SX6036/U1" enhanced port 0 lid 3 lmc 0
[2] "H-0002c903000bf36e"[1](2c903000bf36f)      # "compute060 HCA-1" lid 1 4xQDR
[3] "H-0002c903000bf35a"[1](2c903000bf35b)      # "compute063 HCA-1" lid 28 4xQDR
[4] "H-0002c903000c2646"[1](2c903000c2647)      # "compute062 HCA-1" lid 25 4xQDR
[15]    "S-0002c90200423eb8"[33]        # "Infiniscale-IV Mellanox Technologies" lid 11 4xQDR
[33]    "H-0002c903000c264e"[1](2c903000c264f)      # "compute074 HCA-1" lid 26 4xQDR
[34]    "H-0002c903000c26ee"[1](2c903000c26ef)      # "compute076 HCA-1" lid 48 4xQDR 

So I can't simply rely on there being 36 lines following each switch line or that [36] will always be the last line in a switch block.

Answer 1

Q1

This awk command extract a sorted list of unique computer names from the file, assuming:

The source file is much longer, having a block of lines for each switch.
An script to get a whole switch block (assuming the switch line is always the first line of a continuous set of lines for each switch) sorted and removing repeated nodes is:

awk -v FS='[#"]' '
    BEGIN{c=0}
    $1~/Switch/     {c++; j=0; split($5,arr,"[;:]" ); sw[c,0]=arr[2] }
    $1~/\[[0-9]+\]/ {     j++; split($5,arr," "    ); sw[c,j]=arr[1] }
    END {
            print("final count of switches=" c)
            for (i=1; i<=c; i++) {
                print( "switch=" i, sw[i,0] )     # show switch number.
                split("", out , ":" )             # delete array "out".
                split("", indices , ":" )         # delete array "indices".
                j=0
                while (sw[i,++j]) {               # for all array elements.
                    if (out[sw[i,j]]++ < 1) {     # Is it a new value?
                        indices[sw[i,j]]=j        # add to array "indices".
                    }
                }
                n=asorti(indices)                 # sort the keys of indices
                printf( "%s ", sw[i,0] )
                for (k=1; k<=n; k++) {            # all values for a switch.
                    printf( "%s ", indices[k] )
                }
                printf( "%s\n", "" )
            }
    }
    ' infile

Results:

final count of switches=3
switch=1 ibsw20
ibsw20 Infiniscale-IV compute060 compute061 compute062 compute063
compute064 compute065 compute066 compute067 compute068 compute069
compute070 compute071 compute072 compute073 compute074 compute075
compute076 compute077 
switch=2 ibsw21
ibsw21 Infiniscale-IV compute060 compute061 compute062 compute063
compute064 compute065 compute066 compute067 compute068 compute069
compute070 compute071 compute072 compute073 compute074 compute075
compute076 compute077 
switch=3 ibsw22
ibsw22 Infiniscale-IV compute060 compute062 compute063 compute074 
compute076

I am not sure if Infiniscale-IV should be removed, and if you are also asking for the additional processing needed to get:

SwitchName=ibsw20 Nodes=compute[060-077] 

Q2

From the "man awk":

If RS is set to the null string, then records are separated by blank lines.

That is the "record separator" (RS) set to null:

awk -v RS='' 'script to process lines' file

Answer 2

This is basically BinaryZebra's answer, modified to create a slurm topology.conf file as defined

ibnetdiscover | awk -v FS='[#"]' '
    BEGIN{c=0}
    $1~/Switch/     {c++; j=0; split($5,arr,"[;:]" ); sw[c,0]=arr[2] }
    $1~/\[[0-9]+\]/ && $2~/^H-/ {     j++; split($5,arr," "    ); sw[c,j]=arr[1] }
    END {
            # print("final count of switches=" c)
            for (i=1; i<=c; i++) {
                printf( "SwitchName=s" i, sw[i,0] )     # show switch number.
                split("", out , ":" )             # delete array "out".
                split("", indices , ":" )         # delete array "indices".
                j=0
                while (sw[i,++j]) {               # for all array elements.
                    if (out[sw[i,j]]++ < 1) {     # Is it a new value?
                        indices[sw[i,j]]=j        # add to array "indices".
                    }
                }
                n=asorti(indices)                 # sort the keys of indices
                # printf( "%s ", sw[i,0] )
                printf ( " Nodes=" )
                for (k=1; k<n; k++) {            # all values for a switch.
                    printf( "%s,", indices[k] )
                }
                printf( "%s\n", indices[n] )
            }
    }
    ' | sed -r '/Nodes=$/d' | awk '{sub(/[0-9]+/, ++i)}1; END{printf( "SwitchName=s%s Switches=s[1-%s]\n", NR+1, NR )}'

If a condensed hostlist is desired, simply modify each Node= line using scontrol show hostlist. The modified final pipe would look like:

| awk -F= '{sub(/[[:digit:]]+/, ++i) ; cmd= "scontrol show hostlist " $3 ; cmd | getline line ; printf( "%s=%s=%s\n" , $1, $2, line ) } END{printf( "SwitchName=s%s Switches=s[1-%s]\n", NR+1, NR )}'