0

I am trying to write a script that takes one file and runs each lines as a string for the awk command which is executed on another file. Here is what I have at this point.

#!/bin/bash

FILE=$1
FILE_TO_SEARCH=$2

exec 4> "FILE_TO_SEARCH"

while read -ru 3 LINE; do
    awk -v RS='' -v ORS='\n\n' "$LINE" <&4
done 3< "$FILE"

When I try to run the script I am getting:

./bashscript2.sh: line 8: read: read error: 3: Bad file descriptor

The file (FILE) that it would be searching would have contents like this as an example:

hostAbC
host123
host345
hostMos
hostDef

Which would then run the awk command on a file (FILE_TO_SEARCH) with contents that look like the one below except a lot more.

* * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

 id: urn:storageos:Initiator:
    clustername = BLAHBLAHBLAH
    creationTime = java.util.GregorianCalendar[
                time=1490279415811
                2017-03-23 14:30:15 811ms UTC
,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="UTC",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2017,MONTH=2,WEEK_OF_YEAR=12,WEEK_OF_MONTH=4,DAY_OF_MONTH=23,DAY_OF_YEAR=82,DAY_OF_WEEK=5,DAY_OF_WEEK_IN_MONTH=4,AM_PM=1,HOUR=2,HOUR_OF_DAY=14,MINUTE=30,SECOND=15,MILLISECOND=811,ZONE_OFFSET=0,DST_OFFSET=0]
    host = URI: 
    hostname = hostAbC
    inactive = false
    ininode = 01:01:01:01:01:01:01:01
    iniport = 01:01:01:01:01:01:01:01
    internalFlags = 0
    isManualCreation = true
    label = 01:01:01:01:01:01:01:01
    status = OpStatusMap {}
    protocol = FC
    registrationStatus = REGISTERED


 * * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

 id: urn:storageos:Initiator:
    clustername = YADAYADAYADA
    creationTime = java.util.GregorianCalendar[
                time=1485972630239
                2017-02-01 18:10:30 239ms UTC
,areFieldsSet=true,areAllFieldsSet=true,lenient=true,zone=sun.util.calendar.ZoneInfo[id="UTC",offset=0,dstSavings=0,useDaylight=false,transitions=0,lastRule=null],firstDayOfWeek=1,minimalDaysInFirstWeek=1,ERA=1,YEAR=2017,MONTH=1,WEEK_OF_YEAR=5,WEEK_OF_MONTH=1,DAY_OF_MONTH=1,DAY_OF_YEAR=32,DAY_OF_WEEK=4,DAY_OF_WEEK_IN_MONTH=1,AM_PM=1,HOUR=6,HOUR_OF_DAY=18,MINUTE=10,SECOND=30,MILLISECOND=239,ZONE_OFFSET=0,DST_OFFSET=0]
    host = URI: 
    hostname = hostMos
    inactive = false
    ininode = 01:01:01:01:01:01:01:01
    iniport = 01:01:01:01:01:01:01:01
    internalFlags = 0
    isManualCreation = false
    label = 01:01:01:01:01:01:01:01
    status = OpStatusMap {}
    protocol = FC
    registrationStatus = REGISTERED


* * * * * * * * *  * * * * * * * * * * * * * * * * * * * * * * * * * * * * *

The awk command awk -v RS='' -v ORS='\n\n' would then find the hostAbC entry and return the full entry from space to space.

I can't figure out how to make it work.

  • 1
    Are you sure that's exactly the error you get for exactly that script? Because I can't see any reason read would drop an error there, and I also can't reproduce that error either. (awk will toss an error since the fd it's given to read is opened for writing, but that's not the error you posted here.) – ilkkachu Jan 20 '18 at 19:42
  • 1. is there any particular reason you need to use file descriptors other than stdin and stdout here? 2. running a sh for loop around a bunch of 1-line awk scripts to process the same input file each time is crazy, and will run hundreds (if not many thousands) of times slower than 1 awk script. – cas Jan 21 '18 at 2:43
0

As noted in my comment, unless there's something you haven't told us, there doesn't seem to be any need to use file descriptors other than stdin and stdout here, nor is there any reason not to just pass the filenames to awk on the command line.

And writing a shell while read loop to run awk multiple times against the same input file is a truly awful way to do what you're trying to do - possibly the worst possible way of processing text files. It will be hundreds or thousands of times slower than doing the same task in awk (or sed or perl etc).

Try something like this instead:

#!/bin/bash

FILE1="$1"
FILE_TO_SEARCH="$2"

awk 'NR==FNR { gsub(/([\\.^$(){}\[\]|*+?])/,"\\\\&",$0);
               if (search == "") {
                 search = $0;
               } else {
                 search = search "|" $0;
               };
               next;
             };

     match($0,search)' "$FILE1" RS='' ORS='\n\n' "$FILE_TO_SEARCH"

(line feeds, indentation added for readability. the awk part of this works all squished up on one line too)

This will print out all records in $FILE_TO_SEARCH that match any of the search patterns in $FILE1.

It reads in the first file ($FILE1) using the default RS & ORS, and constructs a regular expression search pattern from it. The gsub() function call is used to backslash-escape all regular expression meta-characters before each line is appended to the search pattern, i.e. all lines are treated as fixed strings. If you want each line to be a regular expression, see the second version below.

With your sample $FILE1 above, the search pattern will be:

hostAbC|host123|host345|hostMos|hostDef 

Then, using RS='' and ORS='\n\n', it reads in the second file ($FILE_TO_SEARCH) and prints any record matching the search pattern.


The following version can be used if you want each line of $FILE1 to be interpreted as a regular expression rather than a fixed string:

#!/bin/bash

FILE1="$1"
FILE_TO_SEARCH="$2"

awk 'NR==FNR { if (search == "") {
                 search = "(" $0 ")" ;
               } else {
                 search = search "|(" $0 ")";
               };
               next;
             };

     match($0,search)' "$FILE1" RS='' ORS='\n\n' "$FILE_TO_SEARCH"

The sample's search pattern with this version would be:

(hostAbC)|(host123)|(host345)|(hostMos)|(hostDef)

Note that with this version it is easily possibly to construct a broken search pattern that either won't match anything or will match too much. You will need to use backslash to escape any regex meta-characters in $FILE1 that you want to be interpreted as literal strings. e.g. if you want to match a literal | then it has to be in the file as \|, otherwise it will be interpreted as a regex OR alternation operator.

  • Thank man! That works but it is only hitting on the last entry of what I have in my $FILE1, it is not looping through all 25. Maybe we need a loop in there? – Juan Aguilera Jan 22 '18 at 16:20
  • that's odd. what output are you getting from the sample data you gave? I'm getting both records (without the lines of * * * *...). The first record matches hostAbC and the second matches hostMos. BTW, there is a loop - every awk script is an implicit loop reading its input file(s)/stdin. – cas Jan 23 '18 at 1:02
1
exec 4> "FILE_TO_SEARCH"

The problem is that you open the file for writing only (deleting its contents by that) but try to use the file descriptor for reading.

exec 4< "$FILE_TO_SEARCH"
  • 1
    There's possibly also a missing $ in front of FILE_TO_SEARCH as the script has a variable by this name. – Kusalananda Jan 20 '18 at 20:08
  • Thank you for looking, I did the changes like you recommended and I am still hitting the same "./bashscript2.sh: line 8: read: read error: 3: Bad file descriptor" – Juan Aguilera Jan 20 '18 at 21:32
  • @JuanAguilera I don't know, works that way with my bash version. You could try opening fd 3 with exec before the loop like with fd 4. Or omitting fd 3 at all. – Hauke Laging Jan 20 '18 at 22:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.