I'm processing data stored into files using a sequence of command line tools.
In order to automate this process, I wrote a bash script chaining theses tools, the result of each tool being fed to the next.

The firsts tools are working with both files and streams, I therefore use bash variables to store temporary results.

    result1=$(tool_1 $filename)
    result2=$(tool_2 <<< "$result1")

The last tool, however, only takes a file names as argument (it is written in C++ and uses the standard std::fstream::open).

To avoid using temporary files, I use the following command, the final result is then sent to the standard output:

    tool_3 /dev/fd/6 6<<<${result2}


While this script seems to work as expected, I randomly encounter problems while running several in parallel (on different files).
It appears that temporary results given to the last tool are corrupted or empty.

Do you see any potential problem with this solution? Did I miss something?

Additional info

Multiple files can be handle by the script. The same idea is used :

    tool_3 /dev/fd/6 /dev/fd/7 6<<<${result20} 7<<<${result21}

After the problem occured, I added a preliminary step to find the first available file descriptor based on (this SO answer) :

    maxfd=$(ulimit -n)
    for fd in $(seq 6 $maxfd); do
        rco="$(true 2>/dev/null >&${fd}; echo $?)"
        rci="$(true 2>/dev/null <&${fd}; echo $?)"
        [[ "${rco}${rci}" = "11" ]] && foundfd=${fd} && break

And finally, I close the descriptors used using:

    for i in $(seq 1 $nfiles); do
      pc="exec $(($i+$foundfd-1))<&-"
      eval $pc

Theses additional steps didn't solve the issue.

  • What if you use tool_3 <(tool_2 <<<"$result1")? (Process substitution) – muru Jul 10 at 0:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.