3

I was testing different methods to produce random garbage and comparing their speed by piping output to pv, as in:

$ cmd | pv -s "$size" -S > /dev/null

I also wanted a "baseline reference", so I measured the the fastest "generator", cat, with the fastest source, /dev/zero:

$ cat /dev/zero | pv -s 100G -S > /dev/null
 100GiB 0:00:33 [2,98GiB/s] [=============================>] 100%   

3GB/s, that's pretty impressive, specially compared to ~70MB I get from /dev/urandom.

But hey, for the special case of /dev/zero I don't need cat! Just for the kicks I removed this textbook UUOC:

$ < /dev/zero pv -s 100G -S > /dev/null
 100GiB 0:00:10 [9,98GiB/s] [=============================>] 100%            

What??? Almost 10GB/s? How can removing cat and a pipe more than triple the speed? If using a slower source such as /dev/urandom the difference is negligible. Is pv doing some voodoo magic? So I tested:

$ dd if=/dev/zero iflag=count_bytes count=200G of=/dev/null status=progress
205392969728 bytes (205 GB, 191 GiB) copied, 16 s, 12,8 GB/s

12,8 GB/s! Same ballpark as pv, and 4 times faster than using pipes.

Is cat to blame? Are pipes so much different than redirection? Afterall, both go to pv as stdin, right? What can explain this huge difference?

1 Answer 1

10

The killer is the use of two processes.

With cat | pv, cat reads and writes, and pv reads and writes, and both processes need to run:

$ perf stat sh -c 'cat /dev/zero | pv -s 100G -S > /dev/null'
 100GiB 0:00:26 [3.72GiB/s] [====================================================================================>] 100%            

 Performance counter stats for 'sh -c cat /dev/zero | pv -s 100G -S > /dev/null':

         34,048.63 msec task-clock                #    1.267 CPUs utilized          
         1,676,706      context-switches          #    0.049 M/sec                  
             3,678      cpu-migrations            #    0.108 K/sec                  
               304      page-faults               #    0.009 K/sec                  
   119,270,941,758      cycles                    #    3.503 GHz                      (74.89%)
   137,822,862,590      instructions              #    1.16  insn per cycle           (74.94%)
    32,379,369,104      branches                  #  950.974 M/sec                    (75.14%)
       216,658,446      branch-misses             #    0.67% of all branches          (75.04%)

      26.865741948 seconds time elapsed

       1.257950000 seconds user
      38.893870000 seconds sys

With pv only, there’s just pv reading and writing, no context switching needed (or hardly any):

$ perf stat sh -c '< /dev/zero pv -s 100G -S > /dev/null'
 100GiB 0:00:07 [13.3GiB/s] [====================================================================================>] 100%            

 Performance counter stats for 'sh -c < /dev/zero pv -s 100G -S > /dev/null':

          7,501.68 msec task-clock                #    1.000 CPUs utilized          
                37      context-switches          #    0.005 K/sec                  
                 0      cpu-migrations            #    0.000 K/sec                  
               198      page-faults               #    0.026 K/sec                  
    27,916,420,023      cycles                    #    3.721 GHz                      (75.00%)
    62,787,377,126      instructions              #    2.25  insn per cycle           (74.99%)
    15,361,951,954      branches                  # 2047.801 M/sec                    (75.03%)
        51,741,595      branch-misses             #    0.34% of all branches          (74.98%)

       7.505304560 seconds time elapsed

       1.768600000 seconds user
       5.733786000 seconds sys

There’s some parallelism (“1.267 CPUs utilized”), but it doesn’t make up for the huge difference in the number of context switches.

Things could be worse, considering the data path — in the first case, data seems to flow from the kernel (/dev/zero), to cat, back to the kernel (for the pipe), to pv, to the kernel (/dev/null). In the second, data flows from the kernel, to pv, back to the kernel. But in the first scenario, pv uses splice to copy data from the pipe, avoiding a trip through kernel-owned memory.

1
  • Wow, what a comprehensive answer! Thank you very much, specially for introducing me to this perf stat tool
    – MestreLion
    Sep 4, 2020 at 19:15

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .