How can I implement a circular flow of data among interconnected commands?

Question

I know of two types how commands can be connected with each other:

1. by using a Pipe (putting std-output into std-input of the next command).
2. by using a Tee (splice the output into many outputs).

I do not know if that is all that is possible, so I draw a hypothetical connection type:

How could it be possible to implement a circular flow of data among commands like for instance in this pseudo code, where I use variables instead of commands.:

pseudo-code:

a = 1    # start condition 

repeat 
{
b = tripple(a)
c = sin(b) 
a = c + 1 
}

Answer 1

Circular I/O Loop Implemented with tail -f

This implements a circular I/O loop:

$ echo 1 >file
$ tail -f file | while read n; do echo $((n+1)); sleep 1; done | tee -a file
2
3
4
5
6
7
[..snip...]

This implements the circular input/output loop using the sine algorithm that you mentioned:

$ echo 1 >file
$ tail -f file | while read n; do echo "1+s(3*$n)" | bc -l; sleep 1; done | tee -a file
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
1.75155632167982146959
[..snip...]

Here, bc does the floating point math and s(...) is bc's notation for the sine function.

Implementation of the Same Algorithm Using a Variable Instead

For this particular math example, the circular I/O approach is not needed. One could simply update a variable:

$ n=1; while true; do n=$(echo "1+s(3*$n)" | bc -l); echo $n; sleep 1; done
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
[..snip...]

Answer 2

You can use a FIFO for this, created with mkfifo. Note however that its very easy to accidentally create a deadlock. Let me explain that—take your hypothetical "circular" example. You feed a command's output to its input. There are at least two ways this might deadlock:

1. The command has an output buffer. It's partially filled, but hasn't been flushed (actually written) yet. It'll do so once it fills. So it goes back to reading its input. It'll sit there forever, because the input it's waiting for is actually in the output buffer. And it won't be flushed until it gets that input...

2. The command has a bunch of output to write. It starts writing it, but the kernel pipe buffer fills up. So it sits there, waiting for their to be space in the buffer. That'll happen as soon as it reads its input, i.e., never as its not going to do that until it finishes writing whatever to its output.

That said, here is how you do it. This example is with od, to create a never-ending chain of hex dumps:

mkfifo fifo
( echo "we need enough to make it actually write a line out"; cat fifo ) \ 
    | stdbuf -i0 -o0 -- od -t x1 | tee fifo

Note that eventually stops. Why? It deadlocked, #2 above. You may also notice the stdbuf call in there, to disable buffering. Without it? Deadlocks without any output.

Answer 3

In general I would use a Makefile (command make) and try to map your diagram to makefile rules.

f1 f2 : f0
      command < f0 > f1 2>f2

To have repetitive/cyclic commands, we need to define a iteration policy. With:

SHELL=/bin/bash

a.out : accumulator
    cat accumulator <(date) > a.out
    cp a.out accumulator

accumulator:
    touch accumulator     #initial value

each make will produce one iteration at the time.

Answer 4

You know, I'm not convinced you necessarily need a repetitive feedback loop as your diagrams portray, so much as maybe you could use a persistent pipeline between coprocesses. Then again, it may be there isn't too much of a difference - once you open a line on a coprocess you can implement typical style loops just writing information to and reading information from it without doing anything very out of the ordinary.

In the first place, it would appear that bc is a prime candidate for a coprocess for you. In bc you can define functions that can do pretty much what you ask for in your pseudocode. For example, some very simple functions to do this could look like:

printf '%s()\n' b c a |
3<&0 <&- bc -l <<\IN <&3
a=1; b=0; c=0;
define a(){ "a="; return (a = c+1); }
define b(){ "b="; return (b = 3*a); }
define c(){ "c="; return (c = s(b)); }
IN

...which would print...

b=3
c=.14112000805986722210
a=1.14112000805986722210

But of course, it doesn't last. As soon as the subshell in charge of printf's pipe quits (right after printf writes a()\n to the pipe) the pipe is torn down and bc's input closes and it quits too. That is not nearly as useful as it could be.

@derobert has already mentioned FIFOs as can be had by creating a named pipe file with the mkfifo utility. These are essentially just pipes as well, except the system kernel links a filesystem entry to both ends. These are very useful, but it would be nicer if you could just have a pipe without risking it being snooped in the filesystem.

As it happens, your shell does this a lot. If you use a shell that implements process substitution then you have a very straightforward means of obtaining a lasting pipe - of the kind that you might assign to a backgrounded process with which you can communicate.

In bash, for instance, you can see how the process substitution works:

bash -cx ': <(:)'
+ : /dev/fd/63

You see it really is a substitution. The shell substitutes a value during expansion that corresponds to the path to a link to a pipe. You can take advantage of that - you needn't be constrained to use that pipe only to communicate with whatever process runs within the () substitution itself...

bash -c '
    eval "exec 3<>"<(:) "4<>"<(:)
    cat  <&4 >&3  &
    echo hey cat >&4
    read hiback  <&3
    echo "$hiback" here'

...which prints...

hey cat here

Now I know that different shells do the coprocess thing in different ways - and that there is a specific syntax in bash for setting one up (and probably one for zsh as well) - but I don't know how those things work. I just know that you can use the above syntax to do virtually the same thing without all of the rigmarole in both bash and zsh - and you can do a very similar thing in dash and busybox ash to achieve the same purpose with here-documents (because dash and busybox do here-documents with pipes rather than temp-files as the other two do).

So, when applied to bc...

eval "exec 3<>"<(:) "4<>"<(:)
bc -l <<\INIT <&4 >&3 &
a=1; b=0; c=0;
define a(){ "a="; return (a = c+1); }
define b(){ "b="; return (b = 3*a); }
define c(){ "c="; return (c = s(b)); }
INIT
export BCOUT=3 BCIN=4 BCPID="$!"

...that's the hard part. And this is the fun part...

set --
until [ "$#" -eq 10 ]
do    printf '%s()\n' b c a >&"$BCIN"
      set "$@" "$(head -n 3 <&"$BCOUT")"
done; printf %s\\n "$@"

...which prints...

b=3
c=.14112000805986722210
a=1.14112000805986722210
#...24 more lines...
b=3.92307618030433853649
c=-.70433330413228041035
a=.29566669586771958965

... and it's still running ...

echo a >&"$BCIN"
read a <&"$BCOUT"
echo "$a"

... which just gets me the last value for bc's a rather than calling the a() function to increment it and prints...

.29566669586771958965

It will continue to run, in fact, until I kill it and tear down its IPC pipes...

kill "$BCPID"; exec 3>&- 4>&-
unset BCPID BCIN BCOUT