1

I've asked a very similar question today before, however I have since realised that I need to increase parameters for the command. I edited the command for a another parameter okay, but the next parameter I had less success with and I don't know why. Here is what I'm trying (and failing) to solve.

I need to check info in each line on a specific set of columns in one variable against all lines in two specific columns in another variable using awk, keeping lines in the first variable that meet parameters.

Attempts I have made so far to do this in one powerful awk command have failed. I can obviously do this in an external loop, but it would be very slow as I have 100's of thousands of lines to check. I appreciate any and all help with solving this, and I am always looking to improve my use of awk, so if you have a solution it would be great to have an explanation so I can learn and improve myself.

Here is an example:

  • Lets say I want to print only the lines from ${ListToCheckFrom}, if column 2 is >= and column 3 is <= to the corresponding columns of any one line from ${ListToCheckAgainst}. Additionally column 1 from ${ListToCheckFrom} must be identical to column 1 in ${ListToCheckAgainst}

  • Input example:

ListToCheckFrom="r,2,3
C,22,24
C,12,13
C,51,59
C,15,20
C,13,18"
        
ListToCheckAgainst="C,25,50
C,22,30
C,12,18
C,15,17
C,1,12
C,60,200"

  • Expected output:
C,22,24  
C,12,13
C,15,20
C,13,18
  • What I have tried based from an answer (thanks to @AdminBee) to a simpler previous question I asked today:
awk -F',' 'list=="constraints"{n++; low[n]=$2;high[n]=$3;c[n]=$1;next}
           {for (i=1;i<=n;i++) {if (($1==c[i])&&($2>=low[i]&&$2<=high[i])||($3>=low[i]&&$3<=high[i])) {print;next};}}' list=constraints <(echo "$ListToCheckAgainst") list=check <(echo "$ListToCheckFrom")

I am using Ubuntu.

3
  • Do you need to have your 2 list contents kept in strings or can they be replaced by one file each with same content ?
    – Cbhihe
    May 11 at 20:12
  • According to your first bullet point, the output should consist of line C,22,24 through to line C,13,18 (5 lines in total starting from line 2 of variable: ListToCheckFrom. It is because you wrote "... to the corresponding columns of any one line from ${ListToCheckAgainst}." or did you mean "of the corresponding line of ${ListToCheckAgainst}" ? The desired output you show seems to indicate that was your intent.
    – Cbhihe
    May 11 at 20:19
  • @Cbhihe No because neither 51 or 59 sit between any numbers on a single line from $ListToCheckAgainst
    – Giles
    May 11 at 20:45

1 Answer 1

1

Building on the other answer you got to your previous question:

$ cat tst.sh
#!/usr/bin/env bash

ListToCheckFrom="r,2,3
C,22,24
C,12,13
C,51,59
C,15,20
C,13,18"

ListToCheckAgainst="C,25,50
C,22,30
C,12,18
C,15,17
C,1,12
C,60,200"

awk '
    BEGIN { FS="," }
    NR==FNR {
        vals[NR] = $0
        next
    }
    {
        for ( nr in vals ) {
            split(vals[nr],v)
            if ( (v[1] == $1) && ( (v[2] <= $2) && ($2 <= v[3]) ) ) {
                print
                next
            }
        }
    }
' <(printf '%s\n' "$ListToCheckAgainst") <(printf '%s\n' "$ListToCheckFrom")

$ ./tst.sh
C,22,24
C,12,13
C,15,20
C,13,18

Regarding the script you asked about modifying, which was this in the answer to your previous question:

awk -F',' 'list=="constr"{n++; low[n]=$2;high[n]=$3;next}
           {for (i=1;i<=n;i++) {if ($2>low[i]&&$2<high[i]) {print;next};}}' \
           list=constr <(echo "$ListToCheckAgainst") \
           list=chk <(echo "$ListToCheckFrom")

The first step for you trying to enhance it should have been to make it a bit easier to read, e.g. if you run the awk part through gawk -o- and add the shell parts back around it you get this:

awk -F',' '
    list == "constr" {
            n++
            low[n] = $2
            high[n] = $3
            next
    }
    
    {
            for (i = 1; i <= n; i++) {
                    if ($2 > low[i] && $2 < high[i]) {
                            print
                            next
                    }
            }
    }
' \
list=constr <(echo "$ListToCheckAgainst") \
list=chk <(echo "$ListToCheckFrom")

and from there hopefully it's easy to see all you had to do to make it work for your new requirements was change it to this:

awk -F',' '
    list == "constr" {
            n++
            key[n] = $1
            low[n] = $2
            high[n] = $3
            next
    }
    
    {
            for (i = 1; i <= n; i++) {
                    if ( (key[i] == $1) && ($2 > low[i] && $2 < high[i]) ) {
                            print
                            next
                    }
            }
    }
' \
list=constr <(echo "$ListToCheckAgainst") \
list=chk <(echo "$ListToCheckFrom")
2
  • This is great Ed, thank you, I don't fully understand it, but it works well and I can try an break it down to learn from it. I tried splitting arrays prior to the first version of this question but I couldn't sort anything that worked. thanks again
    – Giles
    May 11 at 20:57
  • 1
    You're welcome. Feel free to ask questions about any part of it you don't understand.
    – Ed Morton
    May 11 at 21:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.