4

The run times of scripts vary quite much. I would like to find similar time -function as Matlab's timeit, described here.

For instance, I am evaluating these commands here about counting quickly matches, running time LC_ALL=C grep -ao CDA r328_0002.raw | wc -l in loop

---------------------------------------------
Events      real        user        sys
----------- ----------- ----------- ---------
40          0m0.044s    0m0.042s    0m0.005s
40          0m0.064s    0m0.062s    0m0.005s
40          0m0.046s    0m0.044s    0m0.005s
40          0m0.043s    0m0.042s    0m0.005s
40          0m0.047s    0m0.044s    0m0.005s
---------------------------------------------

Table: Events when Macbook Air 2013-Mid in Power Supply. 

---------------------------------------------
Events      real        user        sys
----------- ----------- ----------- ---------
40          0m0.056s    0m0.041s    0m0.011s
40          0m0.060s    0m0.047s    0m0.008s
40          0m0.041s    0m0.039s    0m0.006s
40          0m0.046s    0m0.044s    0m0.006s
40          0m0.047s    0m0.045s    0m0.006s
---------------------------------------------

Table: Events when Macbook Air in Battery Supply, 6h later.  

where you see the real-time fluctuates from 0.044s to 0.064s, the user-time from 0.042s to 0.062s, while the sys-time keeps rather stable at 0.005s. My idea in timing

  • iterate the command first 1k before timing
  • do timing 10 times and take the average and standard deviation

Stout to /dev/nul

This idea is in lcd047's comment, running time LC_ALL=C ggrep -ao CDA r328_0002.raw >/dev/null in loop

--------------------------------------------
real            user            sys
--------------  --------------  ------------
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.008s        0m0.003s        0m0.003s
0m0.006s        0m0.003s        0m0.002s
0m0.005s        0m0.002s        0m0.002s
0m0.006s        0m0.002s        0m0.002s
0m0.009s        0m0.003s        0m0.003s
0m0.007s        0m0.003s        0m0.003s
0m0.006s        0m0.003s        0m0.002s
0m0.006s        0m0.003s        0m0.002s
0m0.008s        0m0.003s        0m0.003s
--------------------------------------------

Table: Events when Macbook Air 2013-Mid in Battery Supply. 

I think these times can be even more improved when keeping the laptop in power supply and keeping less programs on.

How can you time stably shell scripts?

  • 2
    The variations there are just about setup time - all of those are basically the same. Make the sample bigger for more similar looking results: i=0; while [ "$((i+=1))" -lt 1000 ]; do cat r*0002.raw;done | time LC_ALL=C grep ... and so on. – mikeserv Jul 1 '15 at 8:09
  • 2
    In addition to the above advise from @mikeserv, do something like time LC_ALL=C grep -ao CDA r328_0002.raw >/dev/null. Console output is always slow, and is highly influenced by what's going on in the rest of the system. Also, if possible run your benchmarks at a time when the system is idle. If you want accuracy do something like 10 runs, drop the shortest and the longest time, and then average the results. And add enough data to make the tests last longer, preferably >10s, the longer the better. – lcd047 Jul 1 '15 at 11:59
6

The answer is: You can not! Linux is no real time system. The idea of UNIX and therefore Linux, too, is to provide minimum answer times, while the system is shared between multiple users and system processes. Depending on when you start the command, you might have to wait for a important system process to give you your share of processor time. Further the file system might buffer your file you read from the disk, but eventually these file system buffer fail to load the data from cache, when another process on your system occupies that. Generally the time a process needs in a Linux system depends on the surrounding entropy of the machine, where ever it is located in time and space in the universe.

You will need a real time system and specially real time tuned commands and a specific amount of resources reserved for you only. You might get somewhere near that with new CGROUP features from newer kernels, where you can reserve a processor, a part of the memory and an own reflection of the underlying file system for your environment.

One of the major timing problems in your example is the way grep and wc reads the inputs. You might be able to make the timing more stable when you copy your file into a ramfs and work there.

  • What do you estimate is the benefit of initramfs in comparison to CGROUP? – Léo Léopold Hertz 준영 Jul 1 '15 at 8:49
  • 1
    I won't do such estimates and I have no experience with CGROUPs but the CGROUP approach is about ten to million times more complex then the ramfs approach. So I would start with a ramfs and check if this fulfills you needs. Afterwards you can check if you can work in some linux container. – ikrabbe Jul 1 '15 at 9:33
  • Are you still sure? - - Can MATLAB's timeit be implemented in Linux? – Léo Léopold Hertz 준영 Dec 30 '16 at 14:20
  • @LéoLéopoldHertz준영 He's still right. There are entirely too many variables involved to achieve the type of consistent behavior you're after, especially with a shell script. MATLAB is subject to all of the timing vagaries of the operating system and will show similar variations, albeit on a different scale. – Blrfl Dec 30 '16 at 15:34
  • 1
    @LéoLéopoldHertz준영 Let me write you an answer. – Blrfl Dec 30 '16 at 15:44
4
+25

To begin with, you need to understand that shell scripts are just a convenient shorthand for typing the commands yourself. Every time you or a script invokes grep, you're not calling a built-in primitive; you're asking the shell to hunt down a program called grep and start it as a new process. Creation of a new process and running its first bit of code is probably the least-predictable thing you can do in any environment that isn't tuned for predictability. That's going to be the case for any computer running a non-real-time operating system. The details of why this is would be fodder for half of a semester class in operating systems, but I can give you an example or two that illustrate why your efforts to precisely measure full program executions aren't going to yield the consistent results you think they should.

The first things most programs do when first given the CPU is generate a page fault so the first page's worth of code can be loaded. If a dozen other programs are doing I/O on the same device where the code lives, how long your program's pages will take to load depends on how far back the requests are in the device's queue. You may think your test program is running solo, but I'd bet dollars to doughnuts it isn't. (It's also worth mentioning that grep is an I/O-heavy program, so how long it takes to read the input varies for many of the same reasons.)

Many operating systems take steps to avoid having redundant copies of the same code resident at the same time as a way to cut down on memory consumption and increase performance. This means that if you start grep and there's another grep running with its first page already resident, the aforementioned page fault never happens and all of that effort is skipped. This decreases the wall-clock run time.

By the time you get around to doing a timeit in MATLAB, the MATLAB process is already running and it's likely jumped through the necessary hoops to load your function before invoking it repeatedly. Invocation happens fairly quickly because it's just calls internally. While there are also a lot of factors that would affect how long it takes timeit to run, they'd apply equally to anything else.

That said, I think the real reason for what you're seeing is that the comparison isn't apples-to-apples.

Unix's time(1) runs the program under test exactly once where MATLAB's timeit times the function you're testing numerous times and returns the median of the results. The documentation suggests that timeit and the tic and toc functions shouldn't be used together, which hints that the latter are used by the former. The docs for tic and toc recommend that if you have code that runs in anything less than 0.1 seconds, you run it many times and calculate an average. What I take from this is that MATLAB's timing is two orders of magnitude less precise than that of time(1) and is intended more for checking how much time you're able to shave off of long-running functions. That and the combination of averaging and a median will, given enough samples, push a lot of variation into a fairly consistent result.

  • Yes, exactly. I want some unix tool to run time many times to replicate timeit function. Running time once is not enough as you point out in your answer. – Léo Léopold Hertz 준영 Dec 30 '16 at 18:29
  • @LéoLéopoldHertz준영 You could write one in less time than it took for me to write this answer. You're already accumulating the data; just do the math. – Blrfl Dec 30 '16 at 18:36
  • No, it is not possible. I think it is not so easy. The timeit algorithm loops through many iterations. Takes the measurement when the average time of iterations is stable. I think there must be something acknowledged about such an algorithm in Unix. There are also aspects which can be unknown that should be taken into account when doing the measurement. I think it is multifactorial process. – Léo Léopold Hertz 준영 Dec 30 '16 at 19:04
  • @LéoLéopoldHertz준영 The source for timeit is right here, and it's not nearly as sophisticated as you might think. (Although they do get bonus points for trying to subtract out as much measurement overhead as possible.) If you're trying to convince yourself that the wall-clock time to start and execute an I/O-bound program on this kind of system should relatively constant, I think you're going to be sorely disappointed. That's about all I can do for you. – Blrfl Dec 30 '16 at 21:32
  • Promising! There must be some projects which use time in iterations for basic statistics, hardware consideration and session consideration (measurement overhead, ...). I know some of them for NVidia GPU but not anything rough enough that would be consistent for relative statistics in different configurations for the system. – Léo Léopold Hertz 준영 Dec 30 '16 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.