# How can I implement a circular flow of data among interconnected commands?

I know of two types how commands can be connected with each other:

1. by using a Pipe (putting std-output into std-input of the next command).
2. by using a Tee (splice the output into many outputs).

I do not know if that is all that is possible, so I draw a hypothetical connection type:

How could it be possible to implement a circular flow of data among commands like for instance in this pseudo code, where I use variables instead of commands.:

``````pseudo-code:

a = 1    # start condition

repeat
{
b = tripple(a)
c = sin(b)
a = c + 1
}
``````

### Circular I/O Loop Implemented with `tail -f`

This implements a circular I/O loop:

``````\$ echo 1 >file
\$ tail -f file | while read n; do echo \$((n+1)); sleep 1; done | tee -a file
2
3
4
5
6
7
[..snip...]
``````

This implements the circular input/output loop using the sine algorithm that you mentioned:

``````\$ echo 1 >file
\$ tail -f file | while read n; do echo "1+s(3*\$n)" | bc -l; sleep 1; done | tee -a file
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
1.75155632167982146959
[..snip...]
``````

Here, `bc` does the floating point math and `s(...)` is bc's notation for the sine function.

### Implementation of the Same Algorithm Using a Variable Instead

For this particular math example, the circular I/O approach is not needed. One could simply update a variable:

``````\$ n=1; while true; do n=\$(echo "1+s(3*\$n)" | bc -l); echo \$n; sleep 1; done
1.14112000805986722210
.72194624281527439351
1.82812473159858353270
.28347272185896349481
[..snip...]
``````

You can use a FIFO for this, created with `mkfifo`. Note however that its very easy to accidentally create a deadlock. Let me explain that—take your hypothetical "circular" example. You feed a command's output to its input. There are at least two ways this might deadlock:

1. The command has an output buffer. It's partially filled, but hasn't been flushed (actually written) yet. It'll do so once it fills. So it goes back to reading its input. It'll sit there forever, because the input it's waiting for is actually in the output buffer. And it won't be flushed until it gets that input...

2. The command has a bunch of output to write. It starts writing it, but the kernel pipe buffer fills up. So it sits there, waiting for their to be space in the buffer. That'll happen as soon as it reads its input, i.e., never as its not going to do that until it finishes writing whatever to its output.

That said, here is how you do it. This example is with `od`, to create a never-ending chain of hex dumps:

``````mkfifo fifo
( echo "we need enough to make it actually write a line out"; cat fifo ) \
| stdbuf -i0 -o0 -- od -t x1 | tee fifo
``````

Note that eventually stops. Why? It deadlocked, #2 above. You may also notice the `stdbuf` call in there, to disable buffering. Without it? Deadlocks without any output.

• thanks, I did not know anything about buffers in this context, do you know some keywords to read more about it ? Commented Mar 30, 2015 at 22:08
• @AbdulAlHazred For buffering of input/output, look up stdio buffering. For the kernel's buffer in a pipe, pipe buffer seems to work. Commented Mar 30, 2015 at 22:20

You know, I'm not convinced you necessarily need a repetitive feedback loop as your diagrams portray, so much as maybe you could use a persistent pipeline between coprocesses. Then again, it may be there isn't too much of a difference - once you open a line on a coprocess you can implement typical style loops just writing information to and reading information from it without doing anything very out of the ordinary.

In the first place, it would appear that `bc` is a prime candidate for a coprocess for you. In `bc` you can `define` functions that can do pretty much what you ask for in your pseudocode. For example, some very simple functions to do this could look like:

``````printf '%s()\n' b c a |
3<&0 <&- bc -l <<\IN <&3
a=1; b=0; c=0;
define a(){ "a="; return (a = c+1); }
define b(){ "b="; return (b = 3*a); }
define c(){ "c="; return (c = s(b)); }
IN
``````

...which would print...

``````b=3
c=.14112000805986722210
a=1.14112000805986722210
``````

But of course, it doesn't last. As soon as the subshell in charge of `printf`'s pipe quits (right after `printf` writes `a()\n` to the pipe) the pipe is torn down and `bc`'s input closes and it quits too. That is not nearly as useful as it could be.

@derobert has already mentioned FIFOs as can be had by creating a named pipe file with the `mkfifo` utility. These are essentially just pipes as well, except the system kernel links a filesystem entry to both ends. These are very useful, but it would be nicer if you could just have a pipe without risking it being snooped in the filesystem.

As it happens, your shell does this a lot. If you use a shell that implements process substitution then you have a very straightforward means of obtaining a lasting pipe - of the kind that you might assign to a backgrounded process with which you can communicate.

In `bash`, for instance, you can see how the process substitution works:

``````bash -cx ': <(:)'
+ : /dev/fd/63
``````

You see it really is a substitution. The shell substitutes a value during expansion that corresponds to the path to a link to a pipe. You can take advantage of that - you needn't be constrained to use that pipe only to communicate with whatever process runs within the `()` substitution itself...

``````bash -c '
eval "exec 3<>"<(:) "4<>"<(:)
cat  <&4 >&3  &
echo hey cat >&4
echo "\$hiback" here'
``````

...which prints...

``````hey cat here
``````

Now I know that different shells do the coprocess thing in different ways - and that there is a specific syntax in `bash` for setting one up (and probably one for `zsh` as well) - but I don't know how those things work. I just know that you can use the above syntax to do virtually the same thing without all of the rigmarole in both `bash` and `zsh` - and you can do a very similar thing in `dash` and `busybox ash` to achieve the same purpose with here-documents (because `dash` and `busybox` do here-documents with pipes rather than temp-files as the other two do).

So, when applied to `bc`...

``````eval "exec 3<>"<(:) "4<>"<(:)
bc -l <<\INIT <&4 >&3 &
a=1; b=0; c=0;
define a(){ "a="; return (a = c+1); }
define b(){ "b="; return (b = 3*a); }
define c(){ "c="; return (c = s(b)); }
INIT
export BCOUT=3 BCIN=4 BCPID="\$!"
``````

...that's the hard part. And this is the fun part...

``````set --
until [ "\$#" -eq 10 ]
do    printf '%s()\n' b c a >&"\$BCIN"
set "\$@" "\$(head -n 3 <&"\$BCOUT")"
done; printf %s\\n "\$@"
``````

...which prints...

``````b=3
c=.14112000805986722210
a=1.14112000805986722210
#...24 more lines...
b=3.92307618030433853649
c=-.70433330413228041035
a=.29566669586771958965
``````

... and it's still running ...

``````echo a >&"\$BCIN"
echo "\$a"
``````

... which just gets me the last value for `bc`'s `a` rather than calling the `a()` function to increment it and prints...

``````.29566669586771958965
``````

It will continue to run, in fact, until I kill it and tear down its IPC pipes...

``````kill "\$BCPID"; exec 3>&- 4>&-
unset BCPID BCIN BCOUT
``````
• Very interesting. Note with recent bash and zsh you do not have to specify the file descriptor, e.g. `eval "exec {BCOUT}<>"<(:) "{BCIN}<>"<(:)` works as well
– Thor
Commented Apr 15, 2019 at 14:13
• Why does the `eval` in `eval 'exec {myfd}<>'<(:)` make this trick work? `exec {myfd}<>"<(:)"` doesn't work. Commented Oct 7, 2021 at 11:37

In general I would use a Makefile (command make) and try to map your diagram to makefile rules.

``````f1 f2 : f0
command < f0 > f1 2>f2
``````

To have repetitive/cyclic commands, we need to define a iteration policy. With:

``````SHELL=/bin/bash

a.out : accumulator
cat accumulator <(date) > a.out
cp a.out accumulator

accumulator:
touch accumulator     #initial value
``````

each `make` will produce one iteration at the time.

• Cute abuse of `make`, but unnecessary: If you use an intermediate file, why not just use a loop to manage it? Commented Mar 30, 2015 at 21:47
• @alexis, makefiles is probably overkill. I am not very comfortable about loops: I miss the notion of clock, halting condition, or a clear example. The initial diagrams remembered me Workflows, and function signatures. For complex diagrams we will end up needing data conections or and makefile typed rules. (this is just an abusive intuition) Commented Mar 30, 2015 at 22:07
• @alexis, and of course, I agree with you. Commented Mar 30, 2015 at 22:11
• I don't think this is abuse - `make` is about macros which is a perfect application here. Commented Mar 31, 2015 at 7:43
• @mikeserv, Yes. And we all know that abusing tools is the underground Magna Carta of Unix :) Commented Mar 31, 2015 at 8:00