4

I have a bash script containing a group of commands in curly braces { ... }. This group contains some initial echo commands and then one loop. At each iteration the loop executes various slow commands (basically with curl and some extra parsing). Each iteration is slow (because of network interaction) but it prints one line (of python code); as far as I can see, there should be no buffering issue coming from the commands themselves because they terminate their job and leave.

The whole group of commands is piped to python -u (I also tried with tail -f in order to check) and obviously the whole loop is executed before anything is read by python -u or tail -f.

I know how to unbuffer (when possible) one command with various tools like stdbuf but I don't think it can help here because it looks like the issue comes from the command-grouping rather than from such or such command.

Any hint?

  • I would still try unbuffering between the close brace and the next command. – glenn jackman Nov 28 '15 at 15:49
  • Agreed. I think the culprit is the pipe, not the braces. – Tom Zych Nov 28 '15 at 16:02
  • @TomZych - nope, it's not the pipe. programs don't typically buffer input - and it is really weird when they do. they buffer output when it is not written to a terminal. – mikeserv Nov 28 '15 at 20:30
  • Command grouping has no impact on any buffering, so you're looking in the wrong place. Show us your code. – Gilles Nov 28 '15 at 23:18
  • Note to future readers: @mikeserv was right, I was wrong. It wasn't the pipe. See my answer below. – Tom Zych Dec 12 '15 at 22:02
4

(Note to future readers: the tone of exasperation here is not for the question, but for the mistakes I made trying to answer it and the multiple edits they entailed.)

Oh, for pity's sake. The problem is in tail -f. This works just fine:

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
        sleep 0.5
        /bin/echo $i
    done;
} | cat
printf 'bye\n'

It's not the pipe, it's not the group. It's tail. As in, chasing our own tails!

So, tail -f failed because it doesn't output right away for some reason. Not sure why python -u is failing, but I don't think it's anything in the script. Maybe try unbuffer with it. Try your script with cat, at least, and verify that it's unbuffered in that case.


Earlier failed attempt intentionally left here so future readers can make sense of the comments.

This script exhibits the same kind of buffering problem you're getting:

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
    sleep 0.5
    printf '%s\n' $i
    done;
} | tail -f
printf 'bye\n'

This one does not. Output inside the group is redirected to stderr, then stderr from the whole group is piped to the command. Since it's stderr, it's unbuffered.

#!/bin/bash
printf 'hi\n'
{
    for i in 1 2 3 4; do
    sleep 0.5
    printf '%s\n' $i 1>&2
    done;
} |& tail -f
printf 'bye\n'

Adapted from Wang HongQin's answer in this question. The difficulty was in finding a way to unbuffer the pipe with braces rather than an explicit command. Had to fiddle around a while to get the redirection working properly.

  • Hummm... You actually see very well the issue. I see the idea (and I also read the original post by Wang HongQin), but after having tried your very own example by using printf 'print %s\n' $i and by replacing tail -f by python -u, it looks like the print statements are getting displayed without being read by python. I don't think the output is really piped, though it looks like it is (since tail -f is supposed to print something on the screen which happens to be the case here). I am pretty sure that tail in your example doesn't get anything. – Thomas Baruchel Nov 28 '15 at 16:47
  • Yes, sorry, stuff was going out via stderr. I realized it after I posted and had to delete while I worked it out. Revision should work. – Tom Zych Nov 28 '15 at 16:48
  • Don't be sorry; I learn much here ;-) But I still don't think that the new version actually works :-( Could you try with python -u and by replacing your four numbers by a valid python print statement? – Thomas Baruchel Nov 28 '15 at 16:50
  • Sigh, no, you're right. I had turned off the sleep for testing and forgot to make sure that output was unbuffered. I have RL stuff now and can't work on this. I'll leave it up, maybe someone else can start from here and work out the bugs. – Tom Zych Nov 28 '15 at 16:56
  • these examples are not at all pertinent to the question, unless your printf is /bin/printf (though i would doubt it even in that case). the shell doesn't do output buffering in the same way most programs do, and sleep doesn't buffer writes at all because it doesn't do any. – mikeserv Nov 28 '15 at 20:33
1

you just have to do:

{   stdbuf -o0 curl ...
    stdbuf -o0 whatever ...
}|  tail -f

...which will work for dynamically linked applications, though i'm pretty sure curl includes its own unbuffer switch of some kind.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.