2

I have a script like this which I use to broadcast commands to several instaces of the postgresql database CLI connected to several different servers. I'm using a hard-coded set of process-substitutions.

#!/bin/bash
# names have been changed to protect the guilty
cred="user=dbadmin password=SECRET"
domain=example.com

tee \
   >( psql -X "host=db1.$domain dbname=db1 $cred" ) \
   >( psql -X "host=db2.$domain dbname=db2 $cred" ) \
   >( psql -X "host=db3.$domain dbname=db3 $cred" ) \
   >( psql -X "host=xdb1.$domain dbname=xdb1 $cred" ) \
   > /dev/null    
wait

what I would like to do is use a for loop to build up an array of substitutions and pass that array to tee, something like this:

tee "${p[@]}" > /dev/null

but when I use a loop I get every item in $p as /dev/fd/63 tee gives me this error for each.

tee: /dev/fd/63: No such file or directory

example non-working code:

p=()
for z in db1 db2 db3 xdb1
do
  p+=( >( psql -X "host=$z.$domain dbname=$z $cred" ) )
done
tee "${p[@]}" > /dev/null

Is there a way to make this work?

  • is there some reason why you need to do this with tee rather than, say, a for loop? – cas Jun 7 '16 at 5:09
  • I like to paste source for stored procedures in, or type in search queries. a for loop would have to be restarted for each action. some actions are a single line of code, others could be hundereds of lines. plus there's 400ms or more latency opening the SSL connection to the other side of the world. – Jasen Jun 7 '16 at 21:20
  • Queries etc can be stored in a variable or temp file and re-used multiple times by a script. Connection latency applies in any case. IMO simple and robust is always better than flashy and fragile. And with a for loop around psql ... & (or a function which does that in a subshell), it's much simpler to keep track of which connections succeeded and which failed....or just keep re-trying until they succeed. I'll add an answer with an example. – cas Jun 7 '16 at 23:31
  • the tee system is interactive, with a loop it will be batched instead. – Jasen Jun 8 '16 at 1:27
  • huh? batched? interactive? what you say makes no sense. there's no restriction in how either version (process subst with tee, or a loop) can be used. – cas Jun 8 '16 at 1:29
2

This is happening because on this line:

  p+=( >( psql -X "host=$z.$domain dbname=$z $cred" ) )

...bash considers the line to be a complete command. When doing process substitution, the STDIN of the substituted process is closed when the command completes.

There are only 2 ways I can see to do this:

  1. eval. Lets not go there.
  2. exec. Lets go there instead:

 

p=()
for z in db1 db2 db3 xdb1
do
  exec {fd}> >(psql -X "host=$z.$domain dbname=$z $cred")
  p+=( $fd )
done
cd /dev/fd && exec tee "${p[@]}" >/dev/null

The {fd}> syntax causes bash to allocate a new file descriptor, and assign its value to $fd, which we then shove into $p.
Now $p is a bunch of file descriptor numbers, which we have to get tee to write into, so we cd to /dev/fd where the file descriptor numbers are actual files, and then invoke the tee.

(There are numerous other ways to skin the cat, but this is the first and most straightforward that sprang to mind)

  • exec!, I never would have thought to look there. in the end I used p+=( /dev/fd/$fd ) and avoided the cd /dev/fd (for no reason that I can defend logically) – Jasen Jun 7 '16 at 21:07
  • It works, but I don't understand it. what does {fd}> mean and why doesn't exec terminate the shell ? is there documentation for this somewhere? – Jasen Jun 7 '16 at 21:24
  • 1
    good idea on the p+=( /dev/fd/$fd ). Such a simple way to avoid the cd :-). Documentation: {fd}> is under paragraph 2 of the Redirections section, and exec is documented under shell builtin commands (see the bit about "If no command is specified..."). – Patrick Jun 7 '16 at 21:34
  • I replaced exec with true and it still worked. by my reading exec is only needed to manipulate stdin etc. – Jasen Jun 7 '16 at 22:05
  • 1
    exec is for more than just STDIN. exec manipulates file descriptors of the shell itself, not it's children (commands you run). Using true happens to work because it is a shell builtin, and so manipulating the file descriptors of true manipulates the file descriptors of the shell. I would not rely upon the behavior though as whether it is supported or not is highly questionable. – Patrick Jun 8 '16 at 4:17
0

Here's a simple example of what I meant in my comment.

A for-loop (with repeat until success for each psql command, run in a backgrounded sub-shell). No need for an array of generated process substitution file-descriptors.

cred='user=dbadmin password=SECRET'
domain='example.com'

tf=$(mktemp)
cat > "$tf"

for z in db1 db2 db3 xdb1 ; do
  ( while ! psql -X "host=$z.$domain dbname=$z $cred" < "$tf" ; do : ; done) &
done > /dev/null

# don't delete the tempfile until all jobs have completed
wait

rm -f "$tf"

Optionally use sleep x rather than : in the while loop, to have a small delay between each retry attempt.

A smarter version would do more inside the while loop, check the exit status and/or grep stderr to figure out the cause of any failure and respond appropriately.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.