I have this little C89 program -- what it does, how it works, and the undefined behaviour aren't important, just that it does some long memory & math operations on just one thread:

x, c, * b, m, t, i, a;

g (n) {

  for (b = malloc(0); c < n; b[c - 1] = x++, t = 1) {
    char s[9];
    for (i = m = 0; i < sprintf(s, "%d", x); m += a, t *= a) a = s[i++] - 48;
    b = m * t ? x % m + x % t ? b : realloc(b, 4 * ++c) : b;
  }
  return b[c - 1];
}


main (j) {
  printf("%d\n", g(--j));
}

Compile it like this: gcc -std=c89 tt.c -o tt -O3.

Then, if I use a shell script to run it in a loop, to get an idea of its running time:

#!/bin/bash
echo "using input $1"
for _ in `seq 1 10`; do
  ( time ./tt $(seq 1 $1) ) 3>&1 1>/dev/null 2>&3 \
    | grep real \
    | cut -f2
  # sleep 5
done

I see output like this:

$ ./tt.sh 50
using input 50
0m0.016s
0m0.008s
0m0.008s
0m0.007s
0m0.007s
0m0.007s
0m0.008s
0m0.008s
0m0.007s
0m0.007s

Or like this:

$ ./tt.sh 34
using input 34
0m0.007s
0m0.004s
0m0.004s
0m0.004s
0m0.005s
0m0.004s
0m0.003s
0m0.003s
0m0.003s
0m0.004s

There is an initial speedup in the real runtime of the program after the first call, and then all subsequent calls run at this fake speedup.

If I uncomment the # sleep 5 line in the shell script, we see these results:

using input 50
0m0.008s
0m0.020s
0m0.018s
0m0.012s
0m0.009s
0m0.006s
0m0.013s
0m0.012s
0m0.009s
0m0.012s
using input 34
0m0.006s
0m0.007s
0m0.004s
0m0.007s
0m0.008s
0m0.003s
0m0.004s
0m0.004s
0m0.005s
0m0.007s

The times appear more expected and accurate, and discrepancies therein must be attributed to the random state of the processor in that moment (i.e. they are small natural variations).

If I want to get an average running time for my program, I should average these numbers, but sleep 5 between each call, while it is the only way I can find to stop this behaviour, adds up to 50 seconds for 10 tests instead of a few seconds for 20 tests.

I've seen this "replay caching" behaviour before with single-threaded programs that do long operations over and over (tight loops), and I understand it's desirable in 99.9% of all cases.

Assuming this isn't a consequence of some Intel Magic™ at the hardware level, is this something the Linux kernel or Bash are doing intentionally, and how can I make it stop?

I'd like reproducible runtimes for my programs, including library loading and paging from "cold start", without sleep 5ing, because the times affected by caching do not represent a cold start each time.

  • Without the sleep 5, what are the perf stats like for the first run compared to the others? This feels like caching, you should sees lots more misses for the first run than for the rest... – Stephen Kitt Dec 1 '16 at 14:22
  • 2
    Why do you want to defeat caching? It is there for a reason. On the contrary, you should discard the first result; it is the other results which are representative for the running time of your program. The first result includes extraneous operations, such as paging in the code of the program and libraries. – AlexP Dec 1 '16 at 15:27
  • @StephenKitt I notice a small decrease in cache misses over consecutive runs but the size of the drop is not proportional to the change in runtime, and I am not experienced with perf enough to process the output in a way pertinent to the question. – cat Dec 1 '16 at 17:32
  • @AlexP I want to time the program from "cold start" every time, including interpreter and library loader startup. – cat Dec 1 '16 at 17:33
  • Probably won't help but try "echo 3 > /proc/sys/vm/drop_caches" before each invocation of tt. – Mark Wagner Dec 1 '16 at 20:43
up vote 1 down vote accepted

As in my comment:

Probably won't help but try "echo 3 > /proc/sys/vm/drop_caches" before each invocation of tt

Questioner's response:

Yes, echo 1 | sudo tee /proc/sys/vm/drop_caches does exactly what I want (echoing 3 instead gives a weird result). You should put that as an answer

Documentation of values:

To free pagecache:
    echo 1 > /proc/sys/vm/drop_caches
To free reclaimable slab objects (includes dentries and inodes):
    echo 2 > /proc/sys/vm/drop_caches
To free slab objects and pagecache:
    echo 3 > /proc/sys/vm/drop_caches

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.