I have a script that's supposed to get the list of files of two directories, get differences and execute some code for certain files.

These are the commands to get the file lists:

list_in=$(find input/ -maxdepth 1 - type f | sed 's/input\///' | sort -u);
list_out=$(find output/ -maxdepth 1 - type f | sed 's/output\///' | sort -u);

I execute the script in the correct directory, so this shouldn't fail. The unprocessed files are determined by

list_todo=$(comm -23 <(echo "$list_in") <(echo "$list_out"));

since the option -23 for comm only prints lines of the first argument, that don't appear in both arguments and doesn't print lines that uniquely appear in the second argument.

However, only occasionally I get an error saying

command substitution: line 3: syntax error near unexpected token `('
command substitution: line 3: `comm -23 <(echo "$list_in") <(echo "$list_out")'

This really puzzles me, since the exact same script worked fine for the last 3 weeks. I'm using this on a cluster, so several processes might execute the script simultaneously. May the error be caused by this?

Update. The script is called with ./script and I've obviously set chmod +x script before.

(Disclaimer: Even though I'm working on a cluster and these first three lines of my script don't include any locking mechanisms: Of course, no file is ever processed twice)

  • try executing the file using bash file.sh or chmod +x file.sh;./file.sh because if you use sh it starts up in a special, POSIX-compliant mode and See here, #22: "process substitution is not available". – harish.venkat Jan 15 '13 at 17:57
  • @harish.venkat thanks for your comment. I've done this with your second suggestion (except for file extension which shouldn't matter afaik). I've updated the question accordingly – stefan Jan 15 '13 at 18:00
  • if you execute by ./file.sh then you have shebang #!/bin/bash just for clarification – harish.venkat Jan 15 '13 at 18:10
  • 1
    Please post the complete file, or at least a real file that exhibit the problem. You can't have #!/bin/bash, the line to set list_in and the line to set list_out all above the line that sets list_todo if that line is line 3. Also, just in case, put a space between the two closing parentheses, it's conceivable that some version of bash may want )) to close (( or $(( and not two separate )s. – Gilles 'SO- stop being evil' Jan 16 '13 at 0:19
  • @Gilles: I've not specified #!bin/bash in the first line, so the excerpts shown in the question are my actual script (except for the directory name input beeing data, but who cares). But as harish.venkat sais: calling with ./script has implicit /bin/bash shebang. The weird thing is that this isn't consistent behaviour. It obviously can't be complete crap since it worked for over 3 weeks now, meaning roughly 20000 successful runs.. – stefan Jan 16 '13 at 0:24

The kernel recognizes certain file formats that it can execute natively. This includes at least one binary format. Additionally, files that begin with #! (shebang) are considered scripts; for example, if a file is located at /path/to/script and begins with #!/bin/bash then the kernel executes /bin/bash /path/to/script arg1 arg2 when you invoke /path/to/script arg1 arg2.

If the kernel doesn't recognize the file format, it returns ENOEXEC (exec format error) from the execve system call. Following a tradition from the ancient days of Unix kernels that didn't have the shebang feature, most programs make a second attempt to execute a program when the first attempt failed with the error ENOEXEC: they try to execute /bin/sh as the script interpreter.

Bash is a notable exception: it runs the script in itself, not in /bin/sh. So your script was working by accident when you invoked it from bash.

If you leave off the shebang line, your script may be executed under /bin/bash or under /bin/sh, depending on what program it was executed from. And /bin/sh evidently doesn't support process substitution on your system. It's probably still bash (the error message looks like the one from bash), but when bash is invoked under the name sh, it goes into POSIX compatibility mode which doesn't support process substitution.

The moral of the story: if you write a bash script, you must put #!/bin/bash on the first line.


The $() and <() construct is bash only. That means that you need your shebang line to be:


Particularly this will not work:


If you need it to work with sh then you can use ` instead of $():

list_in=`find input/ -maxdepth 1 - type f | sed 's/input\///' | sort -u`

You can use mkfifo in place of <().

  • how is this relevant if I'm stating that I'm in fact using bash? – stefan Jan 15 '13 at 20:57
  • @stefan Might some of the machines in your cluster be running an extremely old version of bash that doesn't have process substitution? (IIRC it was introduced in 2.0, so it would have to be very old.) Or might a sysadmin have symlinked some other shell to bash? – Gilles 'SO- stop being evil' Jan 15 '13 at 23:37
  • @Gilles That's an interesting idea. How to check for odd symlinks? It's certainly odd since afaik all nodes should be identical. – stefan Jan 15 '13 at 23:54
  • @stefan bash --version – Gilles 'SO- stop being evil' Jan 15 '13 at 23:54
  • @Gilles Well two randomly selected nodes, it's 3.2.25. But I've send an email to the support team, thanks. – stefan Jan 16 '13 at 0:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.