4

test case: (with a non-root user, as root would disregard the 000 permission...)

#in a clean directory:
[ -f file_1 ] && chmod 600 file_? # for repeat tests...
for i in file_1 file_2 file_3; do
    printf 'A\nB\n' > "$i" 
    # we need at least 1 char : awk/gawk silently skips empty files...
done
chmod 000 file_2
awk '(FNR==1) { print FILENAME }' file_? 
  # tried with : regular (old unixes) awk on AIX. and gawk on Linux.
  # the fatal "permission denied" on file_2 stops [g]awk.

Is there a way to catch that fatal error & continue with the next files?

(otherwise I find it quite worrysome: using awk on a bunch of files has no guarantee to process all of them, as it will fatally exit if any one of them is unreadable)

Please if possible: answer

  • for regular awk,
  • and for gawk
  • and any other relevant awk versions? (nawk ? etc)
3

2 Answers 2

9

With GAWK:

gawk 'BEGINFILE { if (ERRNO) nextfile } (FNR==1) { print FILENAME }' file_?

In the BEGINFILE block, ERRNO is empty if the file was opened successfully, and nextfile can be used to skip to the next and avoid bailing out with an error.

I don’t think this is supported in other implementations of AWK.

Portably, you can iterate over all the arguments, check whether they point to an unreadable file, and if so, remove them from the arguments before AWK starts processing them; the GAWK manual has an example implementation. This is racy however since a file checked using this loop may become unreadable (or vice versa) before AWK gets round to processing it.

5
  • 1
    the GNU AWK manual also has a supposedly portable piece of code to check beforehand if the files can be read: gnu.org/software/gawk/manual/gawk.html#DOCF74 (though that's of course racy)
    – ilkkachu
    Feb 22 at 16:40
  • 1
    Beware that covers files that can't be opened for reading (open() fails), but not files that can't be read (read() fails) Feb 23 at 11:39
  • Thanks, this is quite a nice protection for gawk (which also has the "BEGINFILE" facility, which is sorely missed in regular awk) (with both with important caveats cited by Ikkachu and Stéphane!). Just a nitpick: maybe edit to say gawk '...' instead of awk '...' to further make clear that this is for the gawk implementation? (you say it above, but the code exerpt would be clearer that way, imo) Feb 23 at 13:11
  • @ikkachu: I may go that route, adding some system() calls to test for the file's presence, and it's readability. this will be much slower (forking commands, retrieving results) but this may help avoid Stephane's warning for read() fails. Maybe with a (NR==1) { wait_for_completion() } (but how to write it?) so that the race is just with the first file of the list? (... not sure if ARGV edits can be taken into account after it starting parsing the first ones?). the ARGV testing would begin with setting a ready="false var, and wait_for_completion() would wait for it to be "true" ? Feb 23 at 13:20
  • @ikkachu ... I didn't read Ed's anwer, which is a much better (and portable) way than what I just imagined trying ^^ Feb 23 at 13:23
3

As @StephenKitt and @ilkkachu already pointed out the gawk manual contains some code that will remove unreadable files from ARGV[] in the BEGIN section but that has a race condition between when the file is tested and when awk actually tries to read it's content, which could be much later if the preceding files are large.

I'd use the script in @StephenKitt's answer if you have gawk or the script in the gawk manual otherwise unless you do think you might have that race condition problem as the gawk manual script is clearer, briefer, simpler, more efficient, etc. than what's below and doesn't require a temp file and global variables, but for those who are concerned about that race condition - here's a more complicated script that'll work in any awk and relies on creating a temp file to open immediately before attempting to open any real file and at THAT point tests if the upcoming real file is readable or not.

$ cat skip.awk
function addTmp(        cmd, oArgv, i, j) {
    cmd = "mktemp"
    cmd | getline TmpChkFile
    close(cmd)

    if ( TmpChkFile != "" ) {
        print "" > TmpChkFile
        close(TmpChkFile)

        for (i in ARGV) {
            oArgv[i] = ARGV[i]
        }
        oArgc = ARGC

        ARGC = 1
        for (i = 1; i < oArgc; i++) {
            if ( ! (oArgv[i] ~ /^[a-zA-Z_][a-zA-Z0-9_]*=.*/ \
                    || oArgv[i] == "-" || oArgv[i] == "/dev/stdin") ) {
                # not assignment or standard input so a file name
                ARGV[ARGC] = TmpChkFile
                ArgFileNames[++j] = oArgv[i]
                ArgFileIndices[j] = ++ARGC
            }
            ARGV[ARGC++] = oArgv[i]
        }
    }
}

function rmvTmp() {
    system("rm -f \047" TmpChkFile "\047")
}

function chkTmp(        stderr, line) {
    if ( (FNR == 1) && (FILENAME == TmpChkFile) ) {
        ++TmpFileNr
        if ( (getline line < ArgFileNames[TmpFileNr]) < 0 ) {
            stderr = "cat>&2"
            printf "Warning: skipping unreadable file \"%s\"\n", ArgFileNames[TmpFileNr] | stderr
            close(stderr)
            delete ARGV[ArgFileIndices[TmpFileNr]]
        }
        close(ArgFileNames[TmpFileNr])
        next
    }
}

BEGIN { addTmp() }
END { rmvTmp() }
{ chkTmp() }

If your awk supports multiple -f arguments (e.g. as required by POSIX) or any other way of executing multiple scripts at once (e.g. GNU awk has @include) then you can use that method to include the above along with your actual script (otherwise copy/paste the above into the same file), e.g. given you have a script like:

$ cat tst.awk
FNR == 1 { print FILENAME, $0 }

and files like:

$ ls file_{1..3}
ls: cannot access 'file_2': No such file or directory
file_1  file_3

then with any POSIX awk (and most, if not all, others) you can do:

$ awk -f skip.awk -f tst.awk file_{1..3}
file_1 A
Warning: skipping unreadable file "file_2"
file_3 C

Most of the work above is done in BEGIN which is called once before the first input file is opened to ensure a readable temp file exists in ARGV[] immediately before every real input file, then chkTmp() is called for every line of input but only does something when it's the first (and only) line of the temp file and that something is to try to open the real input file that's coming after it in ARGV[]. Then END just removes the temp file. So the real extra overhead is the call to chkTmp() and it testing FNR==1 for every input line.

I'm creating a temp file instead of using an existing file because there is no file that is guaranteed to exist on all Unix boxes and, even if there were, it'd have to be exactly 1 line long to avoid adding extra overhead of chkTmp() having to read every line of that file since not all awks support nextfile (or we could call that instead of next inside chkTmp()).

7
  • 1
    I think I missed the best idea of your implementation: the BEGIN close is the one that prevent the race condition, calling the pre-processing before awk can open the first file of the (newly edited) ARGV list... I completely overlooked that ^^ Feb 23 at 13:48
  • 1
    I am an idiot: your solution avoids possible delays between that pre-processing/testing of a file in the list, and the actual time awk needs it !! I just understood what you meant. Apologies Feb 23 at 18:33
  • 1
    I think we can delete this whole exchange (I let you read it first ^^) ... I was clearly not thinking about everything, and you did ! The solution is therefore very good and much better than the top-voted one: it avoids delays between the "checking"-time and "reading that file"-time, is portable amongst many (most, even regular!) awk (I will check solaris's old one, which should fail as usual ^^), and easily added as a -f file. Many thanks for that input and sorry for my inacurate reading (I read only during short pauses at work) Feb 23 at 18:36
  • 1
    @StephenKitt not even that because anyone can rename skip.awk to anything they like and there's no portable way I'm aware of to get the name of the script from inside the script (I thought about it and many other possibilities as I REALLY didn't want to have to create a temp file!), all you can get is the name of awk itself out of ARGV[0]. I suppose you could query the process to see the arguments passed to awk but it's getting cludgy and we really want a placeholder file that's just 1 line for efficiency.
    – Ed Morton
    Feb 23 at 18:57
  • 1
    @OlivierDulac No worries, I'll delete my side of the conversation and leave you to clean yours up. Don't bother testing this or anything else with old, broken awk on Solaris as it's, well, old and broken. I wrote an article about that some time ago and I see someone has re-posted it with improved formatting at awk.freeshell.org/oawk.
    – Ed Morton
    Feb 23 at 19:08

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .