Variable not accepting in awk while passing from a file

Question

I am reading a file line-by-line. Each line looks like this:

xxyu: JHYU_IOPI

Each line is passed to awk as below. I want to print the previous line of the matching pattern; I can achieve this with grep and want to know where I made a mistake with awk.

#!/bin/bash
while read i
do
 awk '/$i/{print a}{a=$0}' ver_in.txt
done<in.txt

I also tried this:

#!/bin/bash
while read i
do
 awk -v var="$i" '/var/{print a}{a=$0}' jil.txt
done<in.txt

Edit: using awk after getting suggestion not to use sh read. My input and desired output are shown below:

EDIT 1: edited the input for @Ed Morton awk script as below

Input file: cat file

/* ----------------- AIX_RUN_WATCH ----------------- */ 

insert_job: AIX_RUN_WATCH   job_type: BOX 
owner: root
permission: 
date_conditions: 1
days_of_week: su
start_times: "22:00"
alarm_if_fail: 1
alarm_if_terminated: 1
group: app
send_notification: 0
notification_emailaddress: 


 /* ----------------- AIX_stop ----------------- */ 

 insert_job: AIXstop   job_type: CMD 
 box_name: AIX_RUN_WATCH
 command: ls
 machine: cfg.mc
 owner: root
 permission: 
 date_conditions: 0
 box_terminator: 1
 std_out_file: ">> /tmp/${AUTOSERV}.${AUTO_JOB_NAME}.$(date +%Y%m%d).stdout"
 std_err_file: ">> /tmp/${AUTOSERV}.${AUTO_JOB_NAME}.$(date +%Y%m%d).stderr"
 alarm_if_fail: 1
 alarm_if_terminated: 1
 group: app
 send_notification: 1


 /* ----------------- AIX_start ----------------- */ 

 insert_job: AIX_start   job_type: CMD 
 box_name: AIX_RUN_WATCH
 command: ls
 machine: cfg.mc
 owner: root
 permission: 
 date_conditions: 0
 box_terminator: 1
 std_out_file: ">> /tmp/${AUTOSERV}.${AUTO_JOB_NAME}.$(date +%Y%m%d).stdout"
 std_err_file: ">> /tmp/${AUTOSERV}.${AUTO_JOB_NAME}.$(date +%Y%m%d).stderr"
 alarm_if_fail: 1
 alarm_if_terminated: 1
 group: app

   cat targets
     box_name: AIX_RUN_WATCH

Expected output -

 box_name: AIX_RUN_WATCH
 insert_job: AIX_stop
 insert_job: AIX_start

Answer 1

for the first attempt you need to use double quotes for shell variable expansion then escape the ones for awk $ operator to prevent it from expanding by the shell but be aware that using like this will break awk in case variable $i was contain special character like \, /. [I'm skipping to fix one or more of another issues with your command now].

while read i
do
 awk "/$i/{print a}{a=\$0}" ver_in.txt
done<in.txt

for the second attempt you need to use either regex match or string match against the current line, like using regex match (partial regex match) with:

while read i
do
 awk -v var="$i" '$0 ~ var{print a}{a=$0}' jil.txt
done<in.txt

or string match (full string match) like:

while read i
do
 awk -v var="$i" '$0==var{print a}{a=$0}' jil.txt
done<in.txt

---

now, talking about the commands you are trying to use them as to print previous line of matching pattern, you can do all with awk and left off using the shell loop; here we are doing full string match:

awk 'NR==FNR { str[$0]; next }
($0 in str) && prev!="" { print prev } { prev=$0 }' in.txt ver_in.txt

or doing partial regex match:

awk 'NR==FNR { patt[$0]; next }
{ for(ptrn in patt) if($0 ~ ptrn && prev!="") print prev; prev=$0 }' in.txt ver_in.txt

or doing partial string match:

awk 'NR==FNR { strings[$0]; next }
{ for(str in strings) if(index($0, str) && prev!="") print prev; prev=$0 }' in.txt ver_in.txt

or doing full regex match:

awk 'NR==FNR { patt[$0]; next }
{ for(ptrn in patt) if($0 ~ "^"ptrn"$" && prev!="") print prev; prev=$0 }' in.txt ver_in.txt

Answer 2

You don't need a while read loop for this, and doing text processing in sh is a bad idea (see Why is using a shell loop to process text considered bad practice?).

Instead get your awk script to process both files.

awk 'NR==FNR { re = $0 "|" re ; next}; # append input line and | to re
     FNR == 1 { sub(/\|$/,"",re) };    # remove trailing | on 1st line of 2nd file

     $0 ~ re { print a }; # if the current line matches re, print a
     {a = $0}' in.txt ver_in.txt

While reading the first file (in.txt), it builds up a regular expression in a variable called re by appending each input line and the regex "alternation" (i.e. OR) operator.

When it has finished reading the first file, the first thing it needs to is remove the trailing | from re. This is necessary because re will always end up with | character due to the way it is constructed. If we don't remove it, that trailing | will cause the regex to match against every line of ver_in.txt.

After that, print variable a if the current input line matches the regex in variable re (this will print an empty line if the first line of ver_in.txt matches re - because a is empty. If you don't want that to happen change that line from $0 ~ re {print a} to $0 ~ re && a != "" {print a}).

Then, whether it matches or not, set a=$0.

NOTE: the NR==FNR {... ; next} is a very common awk idiom for handling the first input file in a different manner than the second and subsequent input files. NR is the global line counter for all files being read, and FNR is the line counter for the current file....so if NR==FNR, that means we're reading the first file. The next statement skips to the next input line, preventing the remainder of the awk script from being executed while in the first file.

You didn't provide a complete data sample, so I made my own to test with:

$ cat in.txt 
xxyu: JHYU_IOPI
foo
bar

This in.txt file will cause re to equal bar|foo|xxyu: JHYU_IOPI

BTW, because the awk script is doing a regex match against re, the lines in in.txt are treated as regular expressions, not as fixed text. That means that if you want any regex special characters (like ., |, [ or ] and many others) in in.txt to be treated as literal characters, you'll need to escape them with a backslash....you would have had to do this with your original sh+awk loop too.

$ cat ver_in.txt 
a line 1
xxyu: JHYU_IOPI
b line 3
d line 4
bar
e line 6
f line 7
foo

Output from the awk script above:

a line 1
d line 4
f line 7

Answer 3

Don't use a shell loop to manipulate text, see Why is using a shell loop to process text considered bad practice?. The people who invented shell also invented awk for shell to call to manipulate text.

Using any awk in any shell on every Unix box:

$ cat tst.awk
NR==FNR {
    tgts[$0]
    next
}
$0 in tgts {
    if ( $0 != prevTgt ) {
        print $0
        prevTgt = $0
    }
    print prevLine
}
{ prevLine = $1 FS $2 }

$ awk -f tst.awk targets file
box_name: AIX_RUN_WATCH
insert_job: AIXstop
insert_job: AIX_start

---

Original answer:

awk '
    BEGIN { RS=""; FS="\n" }
    $2 != prev {
        print $2
        prev = $2
    }
    { print $1 }
' file
ght: ertyjk
xxx: rtyuiol
xxx: ertyuikl_fghjk
xxx: qwertyujkl
xxx: rtyuiol_123
ght: YUIOPO
xxx: rtyuiol
xxx: rtyuiopfghj
xxx: dfghjkvbnm
xxx: qzdfghnbvfgh
xxx: qsxcvghuiokmnhgf

See https://www.gnu.org/software/gawk/manual/gawk.html#Multiple-Line for how setting RS to null lets us work with multi-line records, and then setting FS to a newline means each field in such a record is a whole line so we're treating your data as blank-line separated records, each of which contains 2 lines of data.

You mentioned having some other file of ght lines that indicates which should be printed, implying there are other blocks that should not be printed. If you have such a file and it looks like this:

$ cat targets
ght: ertyjk
ght: YUIOPO

and your other input file contains some ght: lines that do not match the above, e.g. see the ght: whatever blocks in the modified input file below:

$ cat file
xxx: rtyuiol
ght: ertyjk

xxx: ertyuikl_fghjk
ght: ertyjk

xxx: qwertyujkl
ght: ertyjk

xxx: rtyuiol_123
ght: ertyjk

xxx: foo
ght: whatever

xxx: bar
ght: whatever

xxx: rtyuiol
ght: YUIOPO

xxx: rtyuiopfghj
ght: YUIOPO

xxx: dfghjkvbnm
ght: YUIOPO

xxx: qzdfghnbvfgh
ght: YUIOPO

xxx: qsxcvghuiokmnhgf
ght: YUIOPO

then the above code would be updated to:

awk '
    BEGIN { FS="\n" }
    NR==FNR {
        tgts[$0]
        next
    }
    $2 != prev {
        if ( inTgts = ($2 in tgts) ) {
            print $2
        }
        prev = $2
    }
    inTgts { print $1 }
' targets RS='' file
ght: ertyjk
xxx: rtyuiol
xxx: ertyuikl_fghjk
xxx: qwertyujkl
xxx: rtyuiol_123
ght: YUIOPO
xxx: rtyuiol
xxx: rtyuiopfghj
xxx: dfghjkvbnm
xxx: qzdfghnbvfgh
xxx: qsxcvghuiokmnhgf