How to safely use gawk's -i option or @include directive?

Question

Using gawk -i inplace some-awk-code some-file (or @include "inplace" within an awk script) to edit a file in-place (or any other extension) is a security vulnerability.

Why?

How do I work around it?

Answer 1

GNU awk has a few extensions over the standard when it comes to specifying the code to run.

While in standard awk, you can only pass the code either as one or more -f filepath where filepath is taken as the path of a file where to read the code from¹ or as the first non-option argument (as in awk -- 'literal code here'), gawk has a few more options:

• -e 'literal code' (or --source 'literal code') like in sed, you can split the code into more than one argument which you can intersperse with -f filepath ones.
• -E filepath (or --exec filepath), same as -f except there can be only one, and whatever follows it not considered for options or variable assignments, only file paths (or - for stdin).
• --file filepath: alias for -f.
• -i filepath (or --include filepath): similar to -f but with some variations in behaviours as noted in the manual.

Now, the problem is that in gawk, filepath in all of the above is not always treated as a file path:

1. if filepath doesn't exist, gawk will try to open the same file with a .awk extension added. That means it may end up interpreting code you didn't intend it to, but that's unlikely to be a problem in practice as the file you intended it to run has not to exist for that to happen. It doesn't do that with --traditional or -W traditional, but then you can't use most of gawk's extensions with that.
2. if filepath does not contain / characters (and is not -), then the awk program is looked up in a $AWKPATH environment variable in a similar way as shells or execvp() look-up slash-less commands in $PATH, and that's including with --posix and with --traditional, for all of -f/-i/-E (and both without and with a .awk extension added as per above).

The second point is at the core of the problem here.

You can find out what the default AWKPATH is with:

$ (unset -v AWKPATH && gawk 'BEGIN{print ENVIRON["AWKPATH"]}')
.:/usr/share/awk

(even though there's no such variable in the ENVIRONment!)

It starts with ., the current working directory, followed by a system location that contains a few extensions shipped with awk or other third party modules for gawk. On this system:

$ ls /usr/share/awk
assert.awk      getlong.awk    intdiv0.awk    ord.awk           rewind.awk
bits2str.awk    getopt.awk     isnumeric.awk  passwd.awk        round.awk
cliff_rand.awk  gettime.awk    join.awk       processarray.awk  shellquote.awk
ctime.awk       group.awk      libintl.awk    quicksort.awk     strtonum.awk
dpkg-awk.awk    have_mpfr.awk  noassign.awk   readable.awk      walkarray.awk
ftrans.awk      inplace.awk    ns_passwd.awk  readfile.awk      zerofile.awk

That means that for -f/-E, if you want the file in the current working directory to be loaded, you need gawk -f ./file, not gawk -f file which could load a file (or file.awk) from somewhere else if there was no file in the current working directory. Like you need ./cmd in shells to run the cmd in the current working directory (except that $PATH for safety reasons usually doesn't contain ., and that gawk will try to load file.awk as seen above).

That also applies for -i, except one usually uses -i to include gawk extensions from a library in which case you do want those to be looked-up in the directories where they're meant to be found and you do want the .awk extension to be added (as those library extensions usually have such an extension).

In gawk -i inplace 'some code' some-file, you do want gawk to find /usr/share/awk/inplace.awk (or wherever inplace.awk is installed on your system), but the problem here is that the default AWKPATH starts with ., so gawk will first look for it in ./inplace and ./inplace.awk.

If you run that in /tmp or any directory that is writeable or has been writeable by others or generally cannot be trusted, you may end up running malicious code.

For instance, run:

echo 'BEGIN{system("reboot")}' > /tmp/inplace

And you'll find that any script that does awk -i inplace whilst their current working directory is /tmp reboots the system!

To work around that:

• Either hardcode the path of the inplace extension by using awk -i /usr/share/awk/inplace.awk instead of awk -i inplace, though you may need to adapt the path to every system or gawk deployment.

• or remove . and all relative path components from $AWKPATH:

  export AWKPATH="$(LC_ALL=C gawk 'BEGIN {
    n = split(ENVIRON["AWKPATH"], dirs, ":")
    for (i = 1; i <= n; i++)
      if (substr(dirs[i], 1, 1) == "/") {
        newawkpath = newawkpath sep dirs[i]
        sep = ":"
      }
    if (newawkpath == "") newawkpath = "/dev/null"
    print newawkpath}')"
  

  Bearing in mind that you will need to use gawk -f ./file or awk -E ./file to load the file in the current working directory (which you should probably do already even without that change to $AWKPATH as seen above). Also note that versions of gawk prior to 4.1.2 looked into the current working directory after looking in $AWKPATH.

  That approach can't be used in a #! /usr/bin/gawk -E script that uses @include though since the $AWKPATH must already be in the environment by the time gawk is started. So if you have a gawk script that uses @include "some-extension" you need to tell your users to change their $AWKPATH or hard code the path of the extension as per above.

• or use perl which has had a -i option for inplace-editing for decades and can do anything awk can do and much more in a much saner syntax² and fewer portability issues. But don't forget the -- in perl -i -ne 'perl code' -- *.txt or you'd introduce code injection vulnerabilities as well (or use ./*.txt; see Security implications of running perl -ne '...' *)!

---

^{¹ except when that filepath is - in which case most awk implementations interpret it as meaning the code is to be read from standard input.}

^{² perl's -M option, which could be seen as an equivalent of gawk's -i looks the Modules up in $PERL5LIB or $PERLLIB, with a default search path (see (unset -v PERL5LIB PERLLIB && perl -le 'print for @INC') that doesn't include . nor any other relative path}

Answer 2

First off, kudos @StephaneChazelas for saying what I keep saying for years now in every forum i write: hands off of sed -i and awk -i inplace!

In addition to what you already said (and which is in parts new to me, this is even more broken than I thought it is):

1. "-in-place"? Not really!

   sed -i and awk -i inplace both pretend to edit "inplace", but they don't. In fact they create a (hidden) temporary file as output and move that at last, overwriting the original. Basically the same what one does using POSIX-confirming variants, but automatically so. This sounds like a good idea, but from "in-place" i would expect the inode-number and the ownership and filemode to be retained. This is not the case! In fact all three properties are changed given the right prerequisites (i.e. a user allowed to write the file but with a different primary group than the file has, directories with sticky bits, ...).

   Now, don't get me wrong: there is no problem with that happening and it would happen the same way if my process were to write to a temporary file and then do the copying itself. But in this case I'd be aware of it and would make sure the file mode, etc.. is corrected after the change. Since this pretends to work in-place chances are users are not aware of this effect.

2. The non-existing temporary file

   The next problem is: if modifying a file and creating a temporary file in the process I will take precautions: there must be enough space to hold the temporary file, I will make sure to remove the temporary file afterwards, etc.. Since I don't know where the temporary file goes (there is nothing about it in the man page, purportedly all happens "in-place") I have no control over it and in case the system crashes mid-script (these things happen) I have no idea I even left some artefact around to eat up disk space.

Answer 3

Additionally, gawk has an AWKLIBPATH variable for which there is a default value if it's not found in the environment. This variable governs where @load "library" will look up library files: Loading Shared Libraries

The default value appears not to use . directory (for the version I have installed), but I suppose that could change.

Answer 4

As far as @include goes, there is my cppawk. It is a small shell script wrapper for awk which lets you use the C preprocessor, with its #include and macros and all.

#include doesn't look in the current directory. It does something better: when the header name is in double quotes, it looks for it in the same directory as the file in which the #include directive is found. This makes it easy to make a cppawk program out of multiple files: the main file easily finds its other files using relative paths in #include "..." directives.

cppawk has a few library headers of its own, but none provide a solution for in-place file editing. The utility would make it easy to reuse a solution.

Here is a low quality prototype:

$ cat file.bak
alpha
bravo
charlie
$ cp file.bak file
$ cppawk '
#include "inplace.h"
{ out(NR, $0) }
' file
$ cat file
1 alpha
2 bravo
3 charlie

Contents of inplace.h:

BEGIN {
  __inplace_tmpfile = "xyz.tmp"
  __inplace_origfile = ARGV[1]
}

END {
  close(__inplace_tmpfile)
  system("mv " __inplace_tmpfile " " __inplace_origfile)
}

#define out(...) print __VA_ARGS__ >  __inplace_tmpfile

Here we need at least the following: a better way to obtain the temporary file, and also to shell-escape the content of ARGV[1] so that we can safely interpolate it into the mv command.

We could have a default implementation of out somewhere which doesn't have the redirection. We then make a habit of using that instead of print in cppawk programs so that when inplace.h is included, code doesn't have to be modified.

We can achieve some of this without preprocessing, since -f can be used to include script material. Instead of an inplace.h header, we prepare an inplace.awk file which has this:

BEGIN {
  inplace = "xyz.tmp"
  __inplace_origfile = ARGV[1]
}

END {
  close(inplace)
  system("mv " inplace " " __inplace_origfile)
}

We have de-anonymized the name of the variable that holds the temp file; it's part of the interface now.

Unfortunately to be able to mix -f inclusion with in-command-line script material, we need the GNU-specific -e option:

$ mv file.bak file
$ awk -f inplace.awk -e '{ print NR, $0 > inplace }' file
$ cat file
1 alpha
2 bravo
3 charlie

There is also the question of how to refer to inplace.awk. Where do we put it and how do we find it? #include doesn't have that problem. If we ship it with the code, it will find it next to itself. If we stick it into cppawk as a library header it will be <inplace.h>; again, no problem. We also have the option of using cppawk --prepro-only to capture the entire "translation unit", which can then be run without requiring the cpp preprocessor.