Using gawk -i inplace some-awk-code some-file (or @include "inplace" within an awk script) to edit a file in-place (or any other extension) is a security vulnerability.


How do I work around it?


4 Answers 4


GNU awk has a few extensions over the standard when it comes to specifying the code to run.

While in standard awk, you can only pass the code either as one or more -f filepath where filepath is taken as the path of a file where to read the code from¹ or as the first non-option argument (as in awk -- 'literal code here'), gawk has a few more options:

  • -e 'literal code' (or --source 'literal code') like in sed, you can split the code into more than one argument which you can intersperse with -f filepath ones.
  • -E filepath (or --exec filepath), same as -f except there can be only one, and whatever follows it not considered for options or variable assignments, only file paths (or - for stdin).
  • --file filepath: alias for -f.
  • -i filepath (or --include filepath): similar to -f but with some variations in behaviours as noted in the manual.

Now, the problem is that in gawk, filepath in all of the above is not always treated as a file path:

  1. if filepath doesn't exist, gawk will try to open the same file with a .awk extension added. That means it may end up interpreting code you didn't intend it to, but that's unlikely to be a problem in practice as the file you intended it to run has not to exist for that to happen. It doesn't do that with --traditional or -W traditional, but then you can't use most of gawk's extensions with that.
  2. if filepath does not contain / characters (and is not -), then the awk program is looked up in a $AWKPATH environment variable in a similar way as shells or execvp() look-up slash-less commands in $PATH, and that's including with --posix and with --traditional, for all of -f/-i/-E (and both without and with a .awk extension added as per above).

The second point is at the core of the problem here.

You can find out what the default AWKPATH is with:

$ (unset -v AWKPATH && gawk 'BEGIN{print ENVIRON["AWKPATH"]}')

(even though there's no such variable in the ENVIRONment!)

It starts with ., the current working directory, followed by a system location that contains a few extensions shipped with awk or other third party modules for gawk. On this system:

$ ls /usr/share/awk
assert.awk      getlong.awk    intdiv0.awk    ord.awk           rewind.awk
bits2str.awk    getopt.awk     isnumeric.awk  passwd.awk        round.awk
cliff_rand.awk  gettime.awk    join.awk       processarray.awk  shellquote.awk
ctime.awk       group.awk      libintl.awk    quicksort.awk     strtonum.awk
dpkg-awk.awk    have_mpfr.awk  noassign.awk   readable.awk      walkarray.awk
ftrans.awk      inplace.awk    ns_passwd.awk  readfile.awk      zerofile.awk

That means that for -f/-E, if you want the file in the current working directory to be loaded, you need gawk -f ./file, not gawk -f file which could load a file (or file.awk) from somewhere else if there was no file in the current working directory. Like you need ./cmd in shells to run the cmd in the current working directory (except that $PATH for safety reasons usually doesn't contain ., and that gawk will try to load file.awk as seen above).

That also applies for -i, except one usually uses -i to include gawk extensions from a library in which case you do want those to be looked-up in the directories where they're meant to be found and you do want the .awk extension to be added (as those library extensions usually have such an extension).

In gawk -i inplace 'some code' some-file, you do want gawk to find /usr/share/awk/inplace.awk (or wherever inplace.awk is installed on your system), but the problem here is that the default AWKPATH starts with ., so gawk will first look for it in ./inplace and ./inplace.awk.

If you run that in /tmp or any directory that is writeable or has been writeable by others or generally cannot be trusted, you may end up running malicious code.

For instance, run:

echo 'BEGIN{system("reboot")}' > /tmp/inplace

And you'll find that any script that does awk -i inplace whilst their current working directory is /tmp reboots the system!

To work around that:

  • Either hardcode the path of the inplace extension by using awk -i /usr/share/awk/inplace.awk instead of awk -i inplace, though you may need to adapt the path to every system or gawk deployment.

  • or remove . and all relative path components from $AWKPATH:

    export AWKPATH="$(LC_ALL=C gawk 'BEGIN {
      n = split(ENVIRON["AWKPATH"], dirs, ":")
      for (i = 1; i <= n; i++)
        if (substr(dirs[i], 1, 1) == "/") {
          newawkpath = newawkpath sep dirs[i]
          sep = ":"
      if (newawkpath == "") newawkpath = "/dev/null"
      print newawkpath}')"

    Bearing in mind that you will need to use gawk -f ./file or awk -E ./file to load the file in the current working directory (which you should probably do already even without that change to $AWKPATH as seen above). Also note that versions of gawk prior to 4.1.2 looked into the current working directory after looking in $AWKPATH.

    That approach can't be used in a #! /usr/bin/gawk -E script that uses @include though since the $AWKPATH must already be in the environment by the time gawk is started. So if you have a gawk script that uses @include "some-extension" you need to tell your users to change their $AWKPATH or hard code the path of the extension as per above.

  • or use perl which has had a -i option for inplace-editing for decades and can do anything awk can do and much more in a much saner syntax² and fewer portability issues. But don't forget the -- in perl -i -ne 'perl code' -- *.txt or you'd introduce code injection vulnerabilities as well (or use ./*.txt; see Security implications of running perl -ne '...' *)!

¹ except when that filepath is - in which case most awk implementations interpret it as meaning the code is to be read from standard input.

² perl's -M option, which could be seen as an equivalent of gawk's -i looks the Modules up in $PERL5LIB or $PERLLIB, with a default search path (see (unset -v PERL5LIB PERLLIB && perl -le 'print for @INC') that doesn't include . nor any other relative path

  • 1
    @EdMorton, that was my original intention prior to post this and fix my answers, but the behaviour here is clearly as designed and documented. The goals of -f (run a script) and -i (load a module) are clearly hard to reconcile without breaking one or the other. Maybe a new -I that doesn't look in the current directory could be an option. It looks like that's another poor API we'll have to live with and navigate around until a better one is introduced. But feel free to bring it up to Arnold, he may have some better idea on how to improve it. For my part, I think I'll just give up on awk. Commented Jun 23, 2023 at 19:02
  • 1
    @EdMorton the whole thing is a bit half baked. See that AWKPATH that gawk pretends is in ENVIRON when it's not really (having a PROCINFO["awkpath"] initialised from an AWKPATH variable might have been a better API). The @include's that can't be done at run time. Those functions added to /etc/profile.d where that doesn't make much sense as profile.d is for the session initialisation, only read by login shells... Commented Jun 23, 2023 at 19:14
  • 1
    @EdMorton, it's not specific to inplace, that applies to any extension like gawk -i shellquote etc. It's just inplace is the most widely used, and the only one I've ever used myself. Thanks for the link to that discussion though. Commented Jun 24, 2023 at 12:11
  • 1
    @EdMorton, I've now raised it on the bug-gawk mailing list. Commented Jun 24, 2023 at 15:16
  • 1
    "I wonder how many non-inplace gawk -is are out there". Not much returned by a github search. I see some assert, readfile, join being used and a few custom ones, but nowhere as common as -i inplace. Commented Jun 24, 2023 at 21:38

First off, kudos @StephaneChazelas for saying what I keep saying for years now in every forum i write: hands off of sed -i and awk -i inplace!

In addition to what you already said (and which is in parts new to me, this is even more broken than I thought it is):

  1. "-in-place"? Not really!

    sed -i and awk -i inplace both pretend to edit "inplace", but they don't. In fact they create a (hidden) temporary file as output and move that at last, overwriting the original. Basically the same what one does using POSIX-confirming variants, but automatically so. This sounds like a good idea, but from "in-place" i would expect the inode-number and the ownership and filemode to be retained. This is not the case! In fact all three properties are changed given the right prerequisites (i.e. a user allowed to write the file but with a different primary group than the file has, directories with sticky bits, ...).

    Now, don't get me wrong: there is no problem with that happening and it would happen the same way if my process were to write to a temporary file and then do the copying itself. But in this case I'd be aware of it and would make sure the file mode, etc.. is corrected after the change. Since this pretends to work in-place chances are users are not aware of this effect.

  2. The non-existing temporary file

    The next problem is: if modifying a file and creating a temporary file in the process I will take precautions: there must be enough space to hold the temporary file, I will make sure to remove the temporary file afterwards, etc.. Since I don't know where the temporary file goes (there is nothing about it in the man page, purportedly all happens "in-place") I have no control over it and in case the system crashes mid-script (these things happen) I have no idea I even left some artefact around to eat up disk space.

  • 2
    It's true that perl/sed's -i or gawk's inplace.awk don't modify the file in place, break hardlinks and symlinks and may lose some meta data, but there's no perfect solution here. The alternative approaches that keep the original files have issues of their own, like the risk of having a file half processed, it not being atomical, and changing a file while it's being used by another process can have nasty side effects. All is all the perl -i approach (that GNU sed or awk emulate) is one of the least bad ones and probably the best as a generic one. Commented Jun 23, 2023 at 19:07
  • 1
    @StéphaneChazelas: the problem is NOT the automatic replacement of the file to change. I agree, this would be a nifty idea, but: don't call it "in-place" then! Call it "automatic-replacement" or whatever. Do, whatever you want, but don't lie to your user! (side remark: This is exactly what is wrong with M$$ products - if they tell you they do "A" chances are they in fact do "B", because they think it is better for you than the "C" you asked for. ) And, first and foremost: DOCUMENT IT! Why is no mention of this in the man pages? Why do I have to use "truss" to debug my tools?
    – bakunin
    Commented Jun 25, 2023 at 18:24
  • that -inplace comes from perl in the 80s and the behaviour is clearly documented there (GNU and FreeBSD seds added something similar only decades later and GNU awk again much later still). See perldoc perlrun. It's still "in place" in that the editing result ends up in a file with the same name. Maybe not the best word, but I for one can't think of a better one. Commented Jun 25, 2023 at 19:03
  • 1
    An in-place, atomic edit of a file is supposed to be achieved by OS support, via a feature sometimes called "atomic supersede". The trade-offs you see are due to having to simulate that in user space. If you are to have atomicity, you cannot write into the same file. If you don't write into the same file, you don't have .. the same file.
    – Kaz
    Commented Jun 26, 2023 at 15:26
  • If you want a real in-place edit (i.e. one that preserves the inode number, permissions, owner, and group), use ed or ex or similar OR, instead of doing something like mv tmpfile original_file (which is typically what -i / --in-place options do on most programs that have the option) you need to do the equivalent of cat tmpfile > original_file ; rm tmpfile.
    – cas
    Commented Aug 23, 2023 at 2:59

Additionally, gawk has an AWKLIBPATH variable for which there is a default value if it's not found in the environment. This variable governs where @load "library" will look up library files: Loading Shared Libraries

The default value appears not to use . directory (for the version I have installed), but I suppose that could change.

  • Yes, AWKLIBPATH seems to be fine in that regard. Commented Jun 25, 2023 at 16:38

As far as @include goes, there is my cppawk. It is a small shell script wrapper for awk which lets you use the C preprocessor, with its #include and macros and all.

#include doesn't look in the current directory. It does something better: when the header name is in double quotes, it looks for it in the same directory as the file in which the #include directive is found. This makes it easy to make a cppawk program out of multiple files: the main file easily finds its other files using relative paths in #include "..." directives.

cppawk has a few library headers of its own, but none provide a solution for in-place file editing. The utility would make it easy to reuse a solution.

Here is a low quality prototype:

$ cat file.bak
$ cp file.bak file
$ cppawk '
#include "inplace.h"
{ out(NR, $0) }
' file
$ cat file
1 alpha
2 bravo
3 charlie

Contents of inplace.h:

  __inplace_tmpfile = "xyz.tmp"
  __inplace_origfile = ARGV[1]

  system("mv " __inplace_tmpfile " " __inplace_origfile)

#define out(...) print __VA_ARGS__ >  __inplace_tmpfile

Here we need at least the following: a better way to obtain the temporary file, and also to shell-escape the content of ARGV[1] so that we can safely interpolate it into the mv command.

We could have a default implementation of out somewhere which doesn't have the redirection. We then make a habit of using that instead of print in cppawk programs so that when inplace.h is included, code doesn't have to be modified.

We can achieve some of this without preprocessing, since -f can be used to include script material. Instead of an inplace.h header, we prepare an inplace.awk file which has this:

  inplace = "xyz.tmp"
  __inplace_origfile = ARGV[1]

  system("mv " inplace " " __inplace_origfile)

We have de-anonymized the name of the variable that holds the temp file; it's part of the interface now.

Unfortunately to be able to mix -f inclusion with in-command-line script material, we need the GNU-specific -e option:

$ mv file.bak file
$ awk -f inplace.awk -e '{ print NR, $0 > inplace }' file
$ cat file
1 alpha
2 bravo
3 charlie

There is also the question of how to refer to inplace.awk. Where do we put it and how do we find it? #include doesn't have that problem. If we ship it with the code, it will find it next to itself. If we stick it into cppawk as a library header it will be <inplace.h>; again, no problem. We also have the option of using cppawk --prepro-only to capture the entire "translation unit", which can then be run without requiring the cpp preprocessor.

  • This has made me realize that a shell escaping function is essential in the Awk toolkit; cppawk should provide that. Any time you use system() with paths, you have a potential injection problem.
    – Kaz
    Commented Jun 26, 2023 at 16:12
  • system("mv " __inplace_tmpfile " " __inplace_origfile) should be system("mv -- \047" __inplace_tmpfile "\047 \047" __inplace_origfile "\047") or similar so the file names are quoted rather than being exposed to the shell for interpretation (and the -- so it can handle file names that start with -, which may not be possible in your code, idk).
    – Ed Morton
    Commented Dec 19, 2023 at 22:02
  • You'd obviously also need to handle the cases of multiple file names and/or the file name being stored in ARGV[2] or some other location, e.g. to handle something like awk '{print FNR, $0}' OFS=":" file1 OFS=";" file2.
    – Ed Morton
    Commented Dec 20, 2023 at 14:53

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .