Does ssh client interfere with named pipes? [duplicate]

Question

Setup: I have a system in bash where I have a single file that spawns off a pipeline of background processes. In stage one of the pipeline, there are multiple processes writing to a single named pipe. In the next stage of the pipeline there is a reader process which reads the named pipe and passes the data off to other worker processes to operate on the data. (See example script below). These workers can return the data back into the stage2 pipeline if it is determined the data isn't ready via communicating with remote resources via https.

Problem: The setup was working great with a non-trivial load until I added a ssh call inside the worker processes in the second stage of the pipeline. Once I did that a lot of the data going through the named pipe started disappearing. This happens even if I significantly reduce the load to only 2% of what was previously working.

I did some reading and I found a few vague references to the ssh client destroying fds so I wasn't sure if this was related.

Environment: I'm currently using bash 5.0.17 on Ubuntu 20.04. I've also tested on an Ubuntu 22.04 system and saw the same behavior.

Simplified script

#!/bin/bash

function stage1_worker()
{
  # Do work including network calls via a 3rd party program which uses python
  flock --exclusive $QUEUE -c "print \"%s\n\" $DATA > $QUEUE"
}

function stage2_reader_v1()
{
  local QUEUE_DATA=""
  while true ;
  do
    # logic
    if read QUEUE_DATA ; then
      stage2_worker $QUEUE_DATA &
    fi
    # logic
  done < $QUEUE
}

function stage2_worker_v1()
{
    local QUEUE_DATA=$1
    local retry_needed=false
    # logic
    # Add back to the stage 2 queue if retry needed (reader rate limits to stage 2 worker to keep process count down)
    if $retry_needed ; then
        flock --exclusive $QUEUE -c "print \"%s\n\" $DATA > $QUEUE"
    fi
}

QUEUE=$(mktemp -u)
mkfifo $QUEUE
trap "rm -f $QUEUE" EXIT

# Launch stage 2 reader in background process
stage2_reader &
# Launch several stage 1 workers...
stage1_worker $ARG &

# wait for all background processes to complete
wait

When I made this change, suddenly my queue started dropping data and items weren't being processed

function stage2_worker_v2()
{
    local QUEUE_DATA=$1
    local retry_needed=false
    # logic
    local IP_FROM_QUEUE_DATA=... #logic
    ssh -o ConnectTimeout=1 -o BatchMode=yes user@$IP_FROM_QUEUE_DATA '<remote command' >/dev/null 2>&1
    if [[ $? -ne 0 ]] ; then
      #handle failure
    fi
    # Add back to the stage 2 queue if retry needed (reader rate limits to stage 2 worker to keep process count down)
    if $retry_needed ; then
        flock --exclusive $QUEUE -c "print \"%s\n\" $DATA > $QUEUE"
    fi
}

In my quest to limit permanently open fds I changed the reader to this which improved throughput, but I was still missing data

function stage2_reader_v1()
{
  local QUEUE_DATA=""
  while true ;
  do
    # logic
    if read -t 1 QUEUE_DATA <> $QUEUE ; then
      stage2_worker $QUEUE_DATA &
    fi
    # logic
  done
}

Removing the ssh call from the stage2_worker_v2 results in all data flowing through the pipeline as expected.

I'd appreciate any understanding of why this is happening and how to mitigate it.

I wrote this simplified script quickly, there may be minor syntax/naming errors that aren't present in my real code.

Answer 1

It shouldn't interfere with the named pipe, but it does interfere with stdin, namely trying to read from it. When you call stage2_worker, stdin is redirected from the pipe, that carries on the to the ssh process.

  while true; do
    if read QUEUE_DATA ; then
      stage2_worker $QUEUE_DATA &
    fi
  done < $QUEUE

function stage2_worker_v2()
{
    ...
    ssh -o ConnectTimeout=1 -o BatchMode=yes user@$IP

Either redirect the stdin of ssh from somewhere else, e.g. ssh ... < /dev/null (or just use ssh -n which does essentially the same), or do the redirection in the while loop so that it doesn't interfere with the stdin of the stuff inside the loop. There's a few variations:

Tell read to read from fd 3, and redirect the input there (Bash):

  while true; do
    if read QUEUE_DATA -u 3; then
      stage2_worker $QUEUE_DATA &
    fi
  done 3< $QUEUE

Same but with another redirection:

  while true; do
    if read QUEUE_DATA <&3; then
      stage2_worker $QUEUE_DATA &
    fi
  done 3< $QUEUE

Or do the redirection from the pipe only for read:

  while true; do
    if read QUEUE_DATA < $QUEUE; then
      stage2_worker $QUEUE_DATA &
    fi
  done 

In any case, you may want to consider quoting those variable expansions, just in case any of your values ever contain whitespace or glob characters. Also, the standard way to declare a function is funcname() while function funcname() is a mix of the ksh style and the standard way, and is only supported in Bash (AFAIR).