Making Bash send this error to std error

Question

While writing some Lua as a backend for my LuaTeX code I noticed the following. For background, here is the Lua code. This version is standard Lua. But you don't really need to understand Lua to understand what the function is doing.

I needed a function that would return both standard output and error to Lua, and hence to TeX.

Since Lua doesn't have a library function that does this, I had to roll one myself.

This version writes standard output to a file, and redirects standard error to standard output. Then it reads them both and returns them. This isn't particularly pretty, but better alternatives are not obvious.

local function exec_cmd(command, stdout_message, stderr_message)
   local filename=os.tmpname()
   local cmd = string.format(command .. " 2>&1 1> %s", filename)
   local pipe = io.popen(cmd)
   local stderr = pipe:read("*all")
   pipe:close()
   local f = assert(io.open(filename, "r"))
   local stdout = f:read("*all")
   f:close()
   os.remove (filename)
   if stdout ~= nil and stdout ~= '' then
      print(string.format(stdout_message, stdout))
   end
   if stderr ~= nil and stderr ~= '' then
      error(string.format(stderr_message, stderr))
   end
end

Today, while in a state of momentary confusion, I wrote the following code as the argument command to the function above.

"wkhtmltopdf searchpath(foo.html) foo.pdf"

In terms of the Lua function given above, this would be something like

exec_cmd("wkhtmltopdf searchpath(foo.html) foo.pdf", "STDOUT FROM WKHTMLTOPDF is %s", "STDERR FROM WKHTMLTOPDF IS %s")

So this command gets passed to the shell as:

wkhtmltopdf searchpath(foo.html) foo.pdf 2>&1 1> /tmp/lua_vyOiay

for some suitable temporary file e.g. /tmp/lua_vyOiay.

This gives the error, if typed directly on the shell (and forgetting about the Lua, which has served its explanatory purpose)

bash: syntax error near unexpected token `('

But the interesting thing about this is that this error doesn't appear to go to standard output. At least, the above command does not collect it, and I wasn't able to collect it any other way.

So, in the interests of bullet-proofing my Lua function, here are my questions.

1. What is going on above with that nonsense command, and why isn't the error being sent to stderr? This is just for my information.
2. What can I do (if anything) to make sure that I collect the stderr string, and interrupt my TeX program as it should be interrupted?

After further discussion, 2 could be accomplished in at least 2 ways.

2a. Since it seems that the Bash shell itself it giving that error, I'd need a way to trap the standard error of the Bash shell itself, within Bash.

2b. A Lua approach of getting at the standard error of the shell, which here is spawned by Lua.

Answer 1

The Lua program basically boils down to running

io.popen("syntax(error) 2>&1 1>outputfile"):read("*all")

which runs the equivalent of

sh -c 'syntax(error) 2>&1 1>outputfile'

io.popen() captures the stdout of the shell it launches, but not the stderr. If there was no syntax error there, the redirections would arrange stderr to the pipe and stdout to the file. But because of the syntax error, the shell doesn't get that far, the redirections aren't processed. (The shell doesn't understand what you're trying to tell it to do.)

To actually capture both the stdout and stderr, one would usually connect both to pipes (or whatever) before launching the external command. E.g. Python's subprocess.Popen() has the stdout and stderr arguments for that. But Lua's standard library is a bit more barren, and it doesn't look like that's directly available. You'd have to write a C module to provide that function.

As mentioned in the comments, wrapping the command in eval would help, since the resulting command line would then be

eval 'syntax(error)' 2>&1 1>outputfile

and the shell would process the redirections before running the eval. But then of course you need to add an additional set of escaping to protect any single quotes you may have in the inner command.