Bash Script to take a file as input and run awk command on another file

Question

I am trying to write a script that takes one file and runs each lines as a string for the awk command which is executed on another file. Here is what I have at this point.

#!/bin/bash

FILE=$1
FILE_TO_SEARCH=$2

exec 4> "FILE_TO_SEARCH"

while read -ru 3 LINE; do
    awk -v RS='' -v ORS='\n\n' "$LINE" <&4
done 3< "$FILE"

When I try to run the script I am getting:

./bashscript2.sh: line 8: read: read error: 3: Bad file descriptor

The file (FILE) that it would be searching would have contents like this as an example:

hostAbC
host123
host345
hostMos
hostDef

Which would then run the awk command on a file (FILE_TO_SEARCH) with contents that look like the one below except a lot more.

* * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

 id: urn:storageos:Initiator:
    clustername = BLAHBLAHBLAH
    creationTime = java.util.GregorianCalendar[
                time=1490279415811
                2017-03-23 14:30:15 811ms UTC
,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="UTC",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2017,MONTH=2,WEEK_OF_YEAR=12,WEEK_OF_MONTH=4,DAY_OF_MONTH=23,DAY_OF_YEAR=82,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=2,HOUR_OF_DAY=14,MINUTE=30,SECOND=15,MILLISECOND=811,ZONE_OFFSET=0,DST_OFFSET=0]
    host = URI: 
    hostname = hostAbC
    inactive = false
    ininode = 01:01:01:01:01:01:01:01
    iniport = 01:01:01:01:01:01:01:01
    internalFlags = 0
    isManualCreation = true
    label = 01:01:01:01:01:01:01:01
    status = OpStatusMap {}
    protocol = FC
    registrationStatus = REGISTERED


 * * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

 id: urn:storageos:Initiator:
    clustername = YADAYADAYADA
    creationTime = java.util.GregorianCalendar[
                time=1485972630239
                2017-02-01 18:10:30 239ms UTC
,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="UTC",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2017,MONTH=1,WEEK_OF_YEAR=5,WEEK_OF_MONTH=1,DAY_OF_MONTH=1,DAY_OF_YEAR=32,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=1,AM_PM=1,HOUR=6,HOUR_OF_DAY=18,MINUTE=10,SECOND=30,MILLISECOND=239,ZONE_OFFSET=0,DST_OFFSET=0]
    host = URI: 
    hostname = hostMos
    inactive = false
    ininode = 01:01:01:01:01:01:01:01
    iniport = 01:01:01:01:01:01:01:01
    internalFlags = 0
    isManualCreation = false
    label = 01:01:01:01:01:01:01:01
    status = OpStatusMap {}
    protocol = FC
    registrationStatus = REGISTERED


* * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

The awk command awk -v RS='' -v ORS='\n\n' would then find the hostAbC entry and return the full entry from space to space.

I can't figure out how to make it work.

Answer 1

As noted in my comment, unless there's something you haven't told us, there doesn't seem to be any need to use file descriptors other than stdin and stdout here, nor is there any reason not to just pass the filenames to awk on the command line.

And writing a shell while read loop to run awk multiple times against the same input file is a truly awful way to do what you're trying to do - possibly the worst possible way of processing text files. It will be hundreds or thousands of times slower than doing the same task in awk (or sed or perl etc).

Try something like this instead:

#!/bin/bash

FILE1="$1"
FILE_TO_SEARCH="$2"

awk 'NR==FNR { gsub(/([\\.^$(){}\[\]|*+?])/,"\\\\&",$0);
               if (search == "") {
                 search = $0;
               } else {
                 search = search "|" $0;
               };
               next;
             };

     match($0,search)' "$FILE1" RS='' ORS='\n\n' "$FILE_TO_SEARCH"

(line feeds, indentation added for readability. the awk part of this works all squished up on one line too)

This will print out all records in $FILE_TO_SEARCH that match any of the search patterns in $FILE1.

It reads in the first file ($FILE1) using the default RS & ORS, and constructs a regular expression search pattern from it. The gsub() function call is used to backslash-escape all regular expression meta-characters before each line is appended to the search pattern, i.e. all lines are treated as fixed strings. If you want each line to be a regular expression, see the second version below.

With your sample $FILE1 above, the search pattern will be:

hostAbC|host123|host345|hostMos|hostDef 

Then, using RS='' and ORS='\n\n', it reads in the second file ($FILE_TO_SEARCH) and prints any record matching the search pattern.

---

The following version can be used if you want each line of $FILE1 to be interpreted as a regular expression rather than a fixed string:

#!/bin/bash

FILE1="$1"
FILE_TO_SEARCH="$2"

awk 'NR==FNR { if (search == "") {
                 search = "(" $0 ")" ;
               } else {
                 search = search "|(" $0 ")";
               };
               next;
             };

     match($0,search)' "$FILE1" RS='' ORS='\n\n' "$FILE_TO_SEARCH"

The sample's search pattern with this version would be:

(hostAbC)|(host123)|(host345)|(hostMos)|(hostDef)

Note that with this version it is easily possibly to construct a broken search pattern that either won't match anything or will match too much. You will need to use backslash to escape any regex meta-characters in $FILE1 that you want to be interpreted as literal strings. e.g. if you want to match a literal | then it has to be in the file as \|, otherwise it will be interpreted as a regex OR alternation operator.

Answer 2

exec 4> "FILE_TO_SEARCH"

The problem is that you open the file for writing only (deleting its contents by that) but try to use the file descriptor for reading.

exec 4< "$FILE_TO_SEARCH"