1

It is useful to be able to call system commands from awk. However, if you try to use shell extended regex, you'll find it doesn't work.

That is because awk calls /bin/sh instead of /bin/bash as you'd expect in linux these days.

How is it possible to get extended regex to work, when calling the system from awk?

3
  • 2
    This question reads like an answer. Could you split it apart so that the Question is a question? It’s fine to self-answer questions, but answers go in the Answer box.
    – Jeff Schaller
    Oct 26, 2017 at 13:57
  • 2
    Reformulate it into a question / answer format. Copy the content of your question, into an answer. You can answer also your own question. And the current question, convert it to a simple question. On this way, you use the Unix SE as a publication site. The voters typically like the self-answered posts.
    – peterh
    Oct 26, 2017 at 22:35
  • Note that, instead of writing an awk conditional to decide between gunzip and cat, one can use gzip -cdfq to print the contents of a file: gzip data will be uncompressed, non-gzip data will be copied verbatim.
    – dhag
    Oct 27, 2017 at 18:27

2 Answers 2

2

I don't see why you need to do so much in bash when awk is perfectly capable:

BEGIN {
    filename[0]="/media/Pan/test-data/The_long_file.gz";
    filename[1]="/media/Pan/test-data/The_long_file";

    for (n=0; n<2; n++) {
        print "Contents  of file: " filename[n];

        if (filename[n] ~ /\.gz$/) {
            command = "gunzip --to-stdout " filename[n]
            while (( command | getline file_contents ) > 0 ) {
                print file_contents
            }
            close(command)
        }
        else {
            while (( getline line < filename[n]) > 0 ) {
                print line
            }
        }
    }
}
1
  • You don't. The example is particularly aimed at anybody who wants to use a shell from awk, but finds that extended regular expressions don't work - the solution to that problem is the one provided. The example given is just that, an example, it isn't supposed to be a good way of doing anything, just an illustration of how to get extended regex working. Oct 29, 2017 at 6:28
2

It is useful to be able to call system commands from awk. However, if you try to use shell extended regex, you'll find it doesn't work.

That is because awk calls /bin/sh instead of /bin/bash as you'd expect in linux these days.

There is a solution that is not too messy. If you need to read information from various files, some of which are compressed, some of which are not, you can use extended regular expressions in awk like this:

BEGIN   {
        filename[0]="/media/Pan/test-data/The_long_file.gz";
        filename[1]="/media/Pan/test-data/The_long_file";
        for ( n=0;n<2;n++)
                {
                print "Contents  of file: " filename[n];
                command="exec /bin/bash -c \"[[ \"" filename[n] "\" =~ .gz ]] \
                &&gunzip --to-stdout " filename[n] "\
                ||cat " filename[n] "\"";
                while (( command | getline file_contents ) > 0 )
                        print file_contents;
                }
        }

This example lists the contents of the same file /media/Pan/test-data/The_long_file twice, once for the compressed version, once for the plain text.

To test the above, copy it to test.awk, create two files, one compressed, one uncompressed, and put their names into filename[0] and [1], and run it:

awk -f test.awk </dev/null

The example itself isn't very useful, I know, but the escape characters and quotes are all in the right places, and the replacement of /bin/sh by /bin/bash works.

I hope this will save somebody the time it took me to get the syntax right.

The code above solves the problem caused by awk calling /bin/sh by using exec to replace /bin/sh. The code that is passed to the shell is:

 exec /bin/bash -c "[[ \"filename\" =~ .gz ]] &&gunzip --to-stdout filename ||cat filename"

The code that bash executes is:

 [[ "filename" =~ .gz ]] &&gunzip --to-stdout filename ||cat filename

The above extended regular expression checks to see if "filename" matches the expression ".gz". If it does, it executes the gunzip. If it does not, it simply cats the file. You could improve the regular expression by replacing "." with ".", so it only matches a ".", and adding a "$", so it only matches it at the end of the line - I didn't do this to preserve clarity.

1
  • 2
    You shouldn't write the answer in the question.
    – 123
    Oct 26, 2017 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.