3

When I try to redirect the output of cut it always seems to be empty. If don't redirect it, the output shows in terminal as expected. This is true for OS X 10.10 and Linux 4.1.6.

This works:

root@karla:~# nc 10.0.2.56 30003
[...] lots of lines [...]

This works:

root@karla:~# nc 10.0.2.56 30003 | cat
[...] lots of lines [...]

This works:

root@karla:~# nc 10.0.2.56 30003 | cut -d, -f 15,16
[...] lots of lines [...]

This doesn't

root@karla:~# nc 10.0.2.56 30003 | cut -d, -f 15,16 | cat
[nothing]

This again DOES

root@karla:~# cat messung1 | cut -d, -f15,16 | cat
[...] lots of lines [...]

This is not limited to cat after cut. grep, tee and standard redirection using > don't work either.

What's wrong there?

  • 3
    Sounds like buffering issue, run cut with stdbuf -oL cut... – Stéphane Chazelas Aug 31 '15 at 15:45
  • Hey, I just learned about new coreutils command. Thanks, Stéphane! – larsks Aug 31 '15 at 15:53
  • @StéphaneChazelas Thanks, this fixes it for Linux. On OSX the problem persists. – Max Ried Aug 31 '15 at 15:57
  • @StéphaneChazelas unfortunately, this didn't improve over cut – Max Ried Aug 31 '15 at 16:26
  • To find the reason for this problem, I recommend to run cut under truss(1) or your local equivalent. Use truss -o o cut ... and later check the content for the file o. – schily Aug 31 '15 at 16:41
9

It's not that much that there's no output as that it's coming in chunks.

Like many programs, when its output is no longer a terminal, cut buffers its output. That is, it only writes data when it has accumulated a buffer-full of it. Typically, something like 4 or 8 KiB though YMMV.

You can easily verify it by comparing:

(echo foo; sleep 1; echo bar) | cut -c2-

With:

(echo foo; sleep 1; echo bar) | cut -c2- | cat

In the first case, cut outputs oo\n and then ar\n one second later, while in the second case, cut outputs oo\nar\n after 1 seconds, that is when it sees the end of its input and flushes its output upon exit.

In your case, since stdin is nc, it would only see the end of its input when the connection is closed, so it would only start outputting anything after it has accumulated 4KiB worth of data to write.

To work around that, several approaches are possible.

  • On GNU or FreeBSD systems, you can use the stdbuf utility that can tweak the buffering behaviour of some commands (it doesn't work for all as it uses a LD_PRELOAD hack to pre-configure the stdio buffering behaviour).

    ... | stdbuf -oL cut -d, -f15,16 | cat
    

    would tell cut to do a line-based buffering on its stdout.

  • some commands like GNU grep have options to affect their buffering. (--line-buffered in the case of GNU grep).

  • you can use a pseudo-tty wrapper to force the stdout of a command to be a terminal. However most of those solutions have some drawbacks and limitations. The unbuffer expect script for instance, often mentioned to address this kind of problem has a number of bugs for instance.

    One that doesn't work too bad is when using socat as:

    ... | socat -u 'exec:"cut -d, -f15,16",pty,raw' -
    
  • You can replace your text utility with a higher-level text processing tool that has support for unbuffered output.

    • GNU awk for instance has a fflush() function to flush its output. So your cut -d, -f15,16 could be written:

      awk -F, -vOFS=, '{print $15,$16;fflush()}'
      
    • if your awk lacks the fflush() function, you can use system("") instead. That's normally the command to execute a command. Here, we would be executing an empty command, but actually using it for the fact that awk flushes its stdout before running the command.

    • or you can use perl:

      perl -F, -lane 'BEGIN{$,=",";$|=1} print @F[14..15]'
      
  • Thanks. On OSX the only thing I can do right now is to wait. – Max Ried Sep 1 '15 at 4:37
  • on OSX you make be able to brew install coreutils to get stdbuf – Pádraig Brady Sep 1 '15 at 8:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.