The way to use `/usr/bin/env sed -f ` in shebang?

Question

Typing /usr/bin/env sed -f in terminal works.

But if use it as a shebang,

#!/usr/bin/env sed -f 
s/a/b/

The script will be fail to execute:

/usr/bin/env: sed -f: No such file or directory

I kind of believe that it's related with the -f. But how to resolve this?

Answer 1

You can't, portably, put more than one argument on a #! line. That means only a full path and one argument (e.g. #!/bin/sed -f or #!/usr/bin/sed -f), or #!/usr/bin/env and no argument to the interpreter.

A workaround to get a portable script is to use #!/bin/sh and a shell wrapper, passing the sed script as a command-line argument. Note that this is not sanctioned by POSIX (multi-instruction scripts must be written with a separate -e argument for each instruction for portability), but it works with many implementations.

#!/bin/sh
exec sed '
s/a/b/
' "$@"

For a long script, it may be more convenient to use a heredoc. An advantage of a heredoc is that you don't need to quote the single quotes inside, if any. A major downside is that the script is fed to sed on its standard input, with two annoying consequences. Some versions of sed require -f /dev/stdin instead of -f -, which is a problem for portability. Worse, the script can't act as a filter, because the standard input is the script and can't be the data.

#!/bin/sh
exec sed -f - -- "$@" <<'EOF'
s/a/b/
EOF

The downside of the heredoc can be remedied by a useful use of cat. Since this puts the whole script on the command line again, it's not POSIX-compliant, but largely portable in practice.

#!/bin/sh
exec sed "$(cat <<'EOF')" -- "$@"
s/a/b/
EOF

Another workaround is to write a script that can be parsed both by sh and by sed. This is portable, reasonably efficient, just a little ugly.

#! /bin/sh
b ()
{
x
}
i\
f true; then exec sed -f "$0" "$@"; fi
: ()
# sed script starts here
s/a/b/

Explanations:

• Under sh: define a function called b; the contents don't matter as long as the function is syntactically well-formed (in particular, you can't have an empty function). Then if true (i.e. always), execute sed on the script.
• Under sed: branch to the () label, then some well-formed input. Then an i command, which has no effect because it's always skipped. Finally the () label followed by the useful part of the script.
• Tested under GNU sed, BusyBox and OpenBSD. (You can get away with something simpler on GNU sed, but OpenBSD sed is picky about the parts it skips.)

Answer 2

As of GNU coreutils v8.30, you can do:

#!/usr/bin/env -S sed -f

This feature was added in a recent (2018-04-20) commit to env.c in the GNU coreutils package, which added the -S or --split-string option.

From the env man page:

OPTIONS
-S/--split-string usage in scripts
    The  -S  option allows specifing multiple parameters in a script.
    Running a script named 1.pl containing the following first line:

            #!/usr/bin/env -S perl -w -T

    Will execute perl -w -T 1.pl .

    Without the '-S' parameter the script will likely fail with:

            /usr/bin/env: 'perl -w -T': No such file or directory

    See the full documentation for more details.

More examples are available in the GNU coreutils manual.

If you also use the -v option for verbose output, you can see exactly how env splits the string of arguments:

In my_sed_script.sed:

#!/usr/bin/env -vS sed -f
s/a/b/

Execution:

$ ./my_sed_script.sed
split -S:  ‘sed -f’
 into:    ‘sed’
     &    ‘-f’
executing: sed
   arg[0]= ‘sed’
   arg[1]= ‘-f’
   arg[2]= ‘./my_sed_script.sed’

Note: This only applies to shebangs that use /usr/bin/env, as --split-string is a feature of GNU env specifically.

Answer 3

There are various incompatible implementations of the shebang (#!) depending on the OS. Some are building a full argument list, some are preserving the command path and put all remaining arguments as a single one, some are ignoring all of the arguments and pass only the command path, and finally, some are passing the whole string as a single command. You seem to be in the latter case.

Answer 4

env is trying to find a file with name "sed -f". You can try "#!/usr/bin/sed -f" as your shebang line.

Answer 5

This answer provides a path to an elegant solution: https://stackoverflow.com/a/1655389/642372

1. read a heredoc into a shell variable.
2. Pass that variable as a positional argument to sed.

Sample:

#!/usr/bin/env bash

read -rd '' SED_SCRIPT <<EOD      
# Your sed script goes here.
s/a/b/
EOD

exec sed "$SED_SCRIPT" "$@"

Answer 6

I like Walker's solution, but it can be improved (putting this in a separate answer because comments don't accept preformatted text). This may not work on all versions of Linux or Bash, but on Ubuntu 17.10 you can cut this down to the following:

#!/usr/bin/env bash
read SED_SCRIPT <<EOD
s/a/b/
EOD
sed "$SED_SCRIPT" "$@"

You can remove the eval and simplify the read command, but you have to get rid of the comment inside the heredoc.

Also, for everything you ever wanted to know about sed but were afraid to ask, there's a very useful tutorial at http://www.grymoire.com/unix/sed.html. This suggests an even better solution, at the expense of losing the /usr/bin/env and hard-coding the path to sed:

#!/bin/sed -f
s/a/b/
s/c/d/

This has the added advantage of supporting more than one s/// replacement.