3

I have a Wifi antenna and I want to optimise it's position. For that propose I'd like to visualise the current latency in real time.

I wrote a one-line script to output the ping latency:

lat.sh:

ping 8.8.8.8|gawk '/64 bytes/{ match($7,/[0-9.]+/,arr); print((i++),"  ",arr[0]);}'

I want to plot it with gnuplot:

plot "<lat.sh"

That doesn't work. I found out that lat.sh displays what it should to the terminal, such as:

$ ./lat.sh 
0    47.7
1    25.5
2    15.8
3    16.7

However, it does not work to output it to a pipe. ./lat.sh > outfile or ./lat.sh|tee outfile or ./lat.sh &> outfile do not print anything to the terminal, but the file (or pipe) stays empty. Same for the gnuplot pipe. I'm really confused.

4

As mikeserv explains below, this is a buffering issue. One way of solving it would be to force awk to write immediately using fflush() (the rest is just a simplified version of what you were doing):

ping 8.8.8.8 | gawk -F'[= ]' '/^64/{print i++,$(NF-1); fflush()}'

The fflush() call will force awk to print output as soon as it is available without waiting for its buffers to fill.

Alternatively, you could just try running ping a specific number of times. For example, 10:

ping  -c 10 8.8.8.8 | gawk ...
  • "plot" is a gnuplot command, "<command" seem to be the right syntax. However, it expects a absolute path, with that it works. Waiting for the command to finish works, but defeats the whole propose. I want to get a visual feedback while i move the anteanna around. – fosp5 Jun 7 '15 at 15:15
  • @fosp5 on my system it fails with that particular output. plot aa and plot < aa are equivalent as far as I can tell. Perhaps you have a different version of plot which supports a different syntax. Anyway, if this answer solved your issue, please take a moment and accept it by clicking on the check mark to the left. That will mark the question as answered and is the way thanks are expressed on the Stack Exchange sites. – terdon Jun 7 '15 at 15:18
  • I'm using the gnuplot function "plot", not a shell to run a binary called "plot". While your answer did help me, it does not resolve the problem to do this in real-time. – fosp5 Jun 7 '15 at 15:27
  • @fosp5 ah, I see! Sorry, I thought you meant plot from plotutils. Can gnuplot do things in realtime? I'm pretty sure that R can't. I think you will have to print to an output file and then plot that. – terdon Jun 7 '15 at 15:36
  • @mikeserv that was my first assumption as well. I expected output to be printed if I let it run long enough. However, several minutes in, tail -f outfile still showed nothing. It must be some kind of buffering but I don't know what/how. That's why I chickened out and just said it's because it doesn't exit. True but incomplete. – terdon Jun 7 '15 at 17:24
2

It appears the problem here is awk. Your program elides output - trims it. This can be a very useful thing to do, but when you do this you reduce the output queued for awk. Most UNIX programs which use the stdio.h (which is most UNIX programs) will buffer their output depending on the type of their output device. When the output device is your terminal, most UNIX programs line-buffer their output - and do a write() per line. When it is not though, (for example when it is a pipe) said programs will buffer to some defined number of bytes per write.

This is standard behavior and is generally a good thing. Every call to write() is a system call - it's a hook into the system kernel's core and most basic functionality. UNIX systems are time-shared - the kernel shares out the CPU's processing time to each requesting process as necessary. And so it pays to be efficient and block your i/o into manageable chunks. That's why applications buffer.

That's what awk is doing. And this problem is conflated in that its filtering of its input reduces its own output to some 8 or 10 bytes every second or so. If awk buffers output into 4kb chunks, (as is common), and it only generates approximately 10 bytes of output per second, it will wind up writing only once every 6 minutes. It's not going to be a very useful application for measuring latency unless you're checking some frequency on the scale of continental drift.

There are ways to handle this. You can, for example, wrap your pipeline in a pty - screen can do this for you and several other applications besides. Possibly there are awk-native options for handling this - but I wouldn't know (even if terdon definitely does). There is GNU's stdbuf tool which injects a libc call for setting the output buffer at program exec time. This is usually quite effective, but fails to be very useful if the program it execs later adjusts its buffer in the course of its execution.

Now I blame awk but ping might be buffering output as well. However - at least on my linux system - when I call it like:

ping -O -n 8.8.8.8 | ...

...this is not the case. In case you're curious, I managed to make it write every output line consistently as soon as one was available using only very basic POSIX-options to standard UNIX commands.

Most simply:

ping -On 8.8.8.8 |
sed -u 's/^64.*=\(.*\) ttl.*=/\1  /' |
more pipeline

...will work with either a GNU, BSD, or AST sed. Also, there is...

ping -On 8.8.8.8 |
sed 's/^64.*=\(.*\) ttl.*=/\1  /w target_file' |
more pipeline

...which will copy all input lines following any possible processing to stdout - and this will most likely be block-buffered. However, it will also immediately write the results of every successful s///ubstitution to target_file. Immediate writes for any/all specified write files is POSIX-spec'd sed behavior - and the same is (almost) true for read files as well. As a slight variation on that theme:

ping -On 8.8.8.8 |
sed 's/^64.*=\(.*\) ttl.*=/\1  /w /dev/fd/1' |
more pipeline

On systems which support addressing file descriptors with /dev links, the above should result in line-buffered output through the standard shell pipeline - at least as far as sed is concerned. For those systems which do not support the /dev/fd links directly, it is yet a popular sed feature to handle /dev/std(in|out|err) anyway - even the old minised does this. Use -n to disable the default stdout output altogether.

Here is a more flexible - and more complicated - solution based on that construct:

mkfifo  /tmp/ipipe    /tmp/opipe
exec 9<>/tmp/opipe 8<>/tmp/ipipe
trap '  printf \\nTTY:STOP\\n >&8' INT
sed -n 's/^64.*=\(.*\) ttl.*=/\1\t/
        /^TTY:START$/,/^TTY:STOP$/{
                /^TTY/d;w /dev/tty
};      /^PIP:START$/,/^PIP:STOP$/{
                /^PIP/d;w /tmp/opipe
}'      <&8     >/dev/null 2>&1 & SEDPID=$!
ping -On 8.8.8.8       >&8 2>&8 & PNGPID=$!
printf \\nPIP:START\\n >&8;rm /tmp/?pipe

Now that sets up a backgrounded dispatcher process that will always write its output line-buffered. It will write either to the file-descriptor it saved when it is was first called and which corresponds to the parent shell's <>&9 descriptor, or it will write to the tty depending on whether you send it the commands:

PIP:START
TTY:START

You can send these like:

printf \\nPIP:START\\n >&8
printf \\nTTY:START\\n >&8

It will handle both simultaneously as you like. The parent shell has a trap configured for sending it the command TTY:STOP when it receives an INT signal - and so just pressing CTRL+C on your keyboard should be enough to make it stop writing to your terminal at any given time. You will need to explicitly tell it to stop writing to a pipe though, which might also be handled in a trap if you liked. You can tell it to stop like:

printf \\nPIP:STOP\\n >&8

sed will always write on a line-buffer - regardless of how it might buffer its standard out. And so with this you can read each line written by ping as soon as it writes it. You can do it by calling up another process to read on file-descriptor 9 - which is the PIP sed writes to when instructed to do so - or you can just have it print straight to the terminal. To call up another process to read from PIP just do:

cat <&9

...or similar. Also note that I explicitly rm the mkfifo created file-system links for the pipes used as soon as all processes that might need one have definitely established their file handles on those pipes. This means that you will not be able to interact with the processes involved via the filesystem - all IPC must be coordinated through the parent shell's descriptors 8 and 9. The PIDs for the backgrounded sed and ping processes are saved in the shell variables $PNGPID and $SEDPID. Either can be addressed by kill.

Now having run the script example above, then doing sending the TTY:START command the results were:

[mikeserv@localhost ~]$ printf \\nTTY:START\\n >&8

[mikeserv@localhost ~]$ 218 24.2 ms
219 21.2 ms
220 23.1 ms
221 21.3 ms
222 21.9 ms
^C

[mikeserv@localhost ~]$

...but both of the processes are still running - sed just isn't writing output anywhere.

ps -Fp "$SEDPID" "$PNGPID"
UID        PID  PPID  C    SZ   RSS PSR STIME TTY      STAT   TIME CMD
mikeserv 31601 28945  0  2143  1712   4 14:22 pts/0    SN     0:00 sed -n s/^64.*q=\(.*\) ttl.*=/\1\
mikeserv 31602 28945  0  2106   740   3 14:22 pts/0    SN     0:00 ping -On 8.8.8.8
0

Unix seems to be really limited when it comes to pipes. It seems I have write the data into a file first in order to do anything meaning-full with it. If i would rewrite the whole thing a virtual terminal might be the solution. But instead of a fancy gui, I will simply stick to the terminal and render an progress bar with awk.

ping -i 0.5  8.8.8.8 | gawk -F'[= ]' '/^64/{ for(i=0;i<($(NF-1)/3);i++) printf(" "); printf("|");  for(i=120;i>$(NF-1);i--) printf(" "); printf("\r");}'

Replace the 120 with the width of your terminal.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.