Pass text with unbalanced quotes as argument on command line

Question

I want to forward the job output from a job run by a third party scheduler. The scheduler allows me to insert the output into a command, e.g. like so:

python script.py --job-output '{joboutput}'

Where {joboutput} becomes the raw job output forwarded by the scheduler.

The issue that I have encountered is that the job output can contain can contain unbalanced single and double quotes (as well as special characters like (, | etc), so either wrapping the joboutput in single or double quotes doesn't work.

I have looked through many similar questions, but have found none that cover this exact scenario. I would highly appreciate any suggestions. Many thanks!

Answer 1

If that command line is preprocessed by the scheduler and it then sends it to a shell for execution (e.g. though sh -c), after doing a simple text replacement of {joboutput} with the actual text, then what you're asking can't really be done. Not on just one line anyway.

It is possible to pass an arbitrary (NUL-terminated) string as a command line argument (up to some maximum length anyway), but inserting the string on the shell command line requires following the shell's syntax/quoting rules.

Basically, the shell has double quotes, single quotes and backslash escapes. Within double quotes, you need to escape a number of characters by prefixing them with backslashes, so you can't put an arbitrary string inside those. Within single quotes, you don't need to escape anything else, but the single quotes themselves need special processing. The usual way is to replace the quotes with '\'', which just closes the quoted string, inserts an escaped single quote, and reopens the quoted string. In any case, there's some characters that need to be treated specially, and no way around it. It has to be that way, as the shell needs some way of determining where the quoted string ends.

So, if you use "{joboutput}" and {joboutput} is replaced with something that contains ", $, \ or `, it'll break; and if you use '{joboutput}' and {joboutput} is replaced with something that contains a ', it'll break.

Some languages might have more verbose quotes, like Python's """/''', which might help in that they'd allow the occasional lone quote or quote pair, but would of course not still be totally general, since the output could contain the same """ or '''.

The shell doesn't have those, though. The nearest thing is probably here-docs, which are delimited by a freely-chosen line. That would help for embedding almost arbitrary strings, but it does require being able to pass multiple lines, and is rather hairy.

If it's possible to use a multiline command, this should work if {joboutput} is replaced by anything that's not END_OF_JOB_OUTPUT barring parsing bugs in the shell related to the here-doc inside a command substitution. You could change the here-doc separator to any other string, one that's unlikely to appear in the output. However, since the data goes through a command substitution here, any trailing newlines in it are lost.

out=$(cat <<'END_OF_JOB_OUTPUT'
{joboutput}
END_OF_JOB_OUTPUT
)
python script.py --job-output "$out"

or without nesting the here-doc in the command substitution:

exec 9<<'END_OF_JOB_OUTPUT'
{joboutput}
END_OF_JOB_OUTPUT
out=$(cat <&9)
exec 9<&-

printf "%s\n" "$out"

---

If the scheduler was able to launch that command directly, without involving the shell, we wouldn't need to care about the shell's syntax. The scheduler would just need to have some smarts to explicitly pass script.py, --job-output and the job output as distinct arguments. But we don't know if it can do that. (Also, in that case, you wouldn't use the quotes around the placeholder.)

Another way that would be easier than the shell shenanigans above would be to pass the string through an environment variable or a file, if that's supported by the scheduler.