4

I've been trying to process the output of find with parallel, which in turn invoked a shell (some textual substitutions were needed). I observed some strange behaviour, which I cannot really explain to myself.

In each directory there are a bunch of files, call them file1.xtc, file2.xtc. Some of them have names such as file1.part0002.xtc, etc. If the file passed from find had the *.part000x.* name, I need to remove the *.part000x.* bit, such that the resultant command is comething like

command -f file1.part0001.xtc -s file1.tpr 

I used find and parallel to that effect but parallel's substitutions (in particular, the {.} bit) are not quite sufficient (they remove the .xtc extension, leaving the .part0001 alone), so here's a command I used to check my output:

find 1st 2nd 3rd -name '*.xtc' -print0 | parallel -0 sh -c 'name=""; name="{.}"; echo {.} ${name%.*}.tpr'

If I use the above command, first declaring name and assigning an empty string to it (or anything else for that matter), the result is

file1.part0001 file1.tpr

as required (those are the names I need to use for my command). If, however, I run this

find 1st 2nd 3rd -name '*.xtc' -print0 | parallel -0 sh -c 'name="{.}"; echo {.} ${name%.*}.tpr'

the result is:

file1.part0001 .tpr

or it behaves as if $name didn't exist.

So the my questions are:

-what is the reason for this behaviour?

-what would be the preferred way of dealing with it?

The first question is more important here, as the method I used above is a workaround, which, while not pretty, works. It is not the first time I needed to do a textual substitution like that and this behaviour continues to baffle me.

Output of sh --version

GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)

output of a newer version of bash that I installed and used instead of sh in the above command (to the same effect) (/usr/local/bin/bash --version)

GNU bash, version 4.2.0(1)-release (i386-apple-darwin11.4.2)
5

Your problem has nothing to do with bash. In fact, since you're telling parallel to run sh, you may not even be using bash.

The issue is that parallel is not really a drop-in replacement for xargs, as its documentation indicates. Instead, it accumulates its arguments into a single string (with spaces between them) and then interprets that as a series of commands. So, in your case, you have:

sh -c 'name="{.}"; echo {.} ${name%.*}.tpr'

which is interpreted as

sh -c 'name="{.}";
echo {.} ${name.*}.tpr

Since those are two separate commands, and the first one executes in a subshell (sh -c), $name is not set in the second one.

Now, you could add pretty well anything to the beginning of the string, such as true:

sh -c 'true; name="{.}"; echo {.} ${name%.*}.tpr'

That will be interpreted as:

sh -c 'true'
name="{.}"
echo {.} ${name%.*}.tpr'

In this case, the call to sh is essentially a throw-away; then name is set in the environment maintained by parallel and finally echo is called with name set.

So it would appear that the easiest solution is simply to get rid of the unnecessary call to sh:

find 1st 2nd 3rd -name '*.xtc' -print0 |
parallel -0 'name={.}; echo {.} "${name%.*}.tpr"'

Note: Based on a hint given by @StephaneChazelas, I removed the quotes around {.} and added them around ${name%.*}.ptr. parallel does its own quoting of its own substitutions, which interferes in some odd ways with explicit quotes. However, it does not add quoting to shell substitutions, which should be quoted if there is any possibility of the substitution being word-split.

Another option, if you really want to use a subshell for some reason (or a particular subshell), would be to use the -q option:

find 1st 2nd 3rd -name '*.xtc' -print0 |
parallel -0 -q sh -c 'name="{.}"; echo "{.}" "${name%.*}.tpr"'

Note: As above, I adjusted the quotes. In this case, the explicit -q suppresses the quoting of substitutions, so you have to quote them explicitly. However, this is a textual quotation, which is less accurate than shell quoting; if the substitution includes a double-quote character, that character will not be escaped, so it will close the explicit quotes, breaking the command line and effectively introducing a command injection vulnerability (you'd get other problems for file names contain $, `, or \ characters). For this, amongst other reasons, the -q option is discouraged.

  • That is a very good point. I was indeed expecting it to behave like xargs (where the first, empty declaration of name was not necessary). I thought the single quotes should sort things out and the QUOTING section of the manpage was much needed in the end to figure out what was going on (again, contrary to expectations). – Wojtek Rzepala Aug 7 '13 at 22:18
  • 1
    @WojtekRzepala. Yes, unfortunately GNU parallel always calls a shell, and quotes the {}, {.}... for the shell. For that reason, you should not quote it yourself (so rici's solution is wrong in that regard), and you should avoid backticks for command substitution. – Stéphane Chazelas Aug 8 '13 at 4:32
  • @StephaneChazelas: OK, I think I worked it out. Feel free to critique if I got it wrong. – rici Aug 8 '13 at 5:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.