1

I am running a benchmark to figure out the number of jobs I should allow GNU Make to use in order to have optimal compile time. To do so, I am compiling Glibc with make -j<N> with N an integer from 1 to 17. I did this 35 times per choice of N so far (35*17=595 times in total). I am also running it with GNU Time to determine the time spent and resources used by Make.

When I was analyzing the resulting data, I noticed something a little peculiar. There is a very noticeable spike in the number of major page faults when I reach -j8.

average number of major page faults

I should also note that 8 is the number of CPU cores (or number of hyper-threads to be more specific) on my computer.

I can also notice the same thing, but less pronounced, in the number of voluntary context switches.

average number of voluntary context switches

To make sure my data wasn't biased or anything, I ran the test two more times and I still get the same result.

I am running artix linux with linux kernel 5.15.12.

What is the reason behind these spikes?

EDIT: I've done the same experiment again on a 4 cores PC. And I can observe the same phenomenon, at the 4 jobs mark this time around.

average number of major page faults on a 4 cores CPU

Also, notice the jump in major page faults in the 2 jobs mark.

average number of voluntary context switches on a 4 cores CPU

EDIT2:

@FrédéricLoyer suggested comparing page faults with the efficiency (inverse of the elapsed time). Here is a box plot of exactly that:

box plot of inverse elapsed real time (wall clock) in 1/seconds

We can see that the efficiency is getting better as we go from 1 job to 4 jobs. But it stays basically the same for bigger numbers of jobs.

I should also mention that my system has enough memory so that even with the maximum number of jobs, I do not run out of memory. I am also recording the PRSS (peak resident set size) and here is a box plot of it.

box plot of peak resident set size in KB

We can see that the number of jobs doesn't impact memory usage at all.

EDIT3: As MC68020 suggested, here are the plots for TLBS (Transaction Lookaside Buffer Shootdown) values for 4 cores and 8 cores systems, respectively:

transaction lookaside buffer shootdown - 4 cores

transaction lookaside buffer shootdown - 8 cores

10
  • Can you disclose the status of your system regarding kernel misc. mitigations in particular (Page Table Isolation) and additionally also provide figures regarding TLBS (Transaction Lookaside Buffer Shootdown) interrupts ?
    – MC68020
    Jan 15 at 9:57
  • Hello @MC68020, Since the first machine (The 8 cores) uses an AMD CPU it doesn't have page-table isolation enabled, while the second (the 4 cores) is an Intel CPU and I believe it has that enabled. The way I checked is by looking at /sys/devices/system/cpu/vulnerabilities/meltdown and it says Mitigation: PTI. Do you think it's safe to say that it is enabled? As for TLBS interrupts, I am not sure how to record those Jan 15 at 21:13
  • @MC68020 please let me know if you know of a way to record TLBS interrupts Jan 15 at 21:49
  • grep TLB /proc/interrupts. Capture the total count immediately prior to each make and the total count immediately after each make completion. Record the difference. (No need to repeat that 35 times.)
    – MC68020
    Jan 15 at 21:52
  • 1
    @MC68020 on the Intel machine /sys/devices/system/cpu/smt/control says on. I also added plots for TLBS values Jan 16 at 21:20
1
+50

Because your graph showing the global efficiency provides the correct answer to your quest, I'll try to focus on explanations.

A/ EFFICIENCY \ JOBS PLACEMENT

Theoretically, (Assuming all CPUs idle at make launch time and no other task running and no i job has already completed when launching the n-th > i), we may expect CFS to distribute the 1,2,3,4,5,6,7,8 jobs to CPU 0,2,4,6 (because no benefits from cache sharing) then 1,3,5,7 (still no benefits from cache sharing but because of cache being shared between siblings, increase of lock contention hence negative impact on global efficiency) Could this be enough to explain the lack of improvement of global efficiency starting from job 5 ?

B/ PAGE FAULTS

As explained by Frédéric Loyer, major page faults are expected at job launch time (due to the necessary read system calls). Your graph shows the increase is almost constant from 5 to 8 jobs. The significant increase at -j4 on your 4+4 core (corroborated by the significant increase at -j2 on your 2+2 core) appears to me more intriguing. Could this be the witness of the rescheduling of one job's thread on whatever > 4 cpu because of whatever sudden activity of some <=4 cpu caused by whatever other task ? The constant amount of page faults for -j(n>8) being explained by the fact that all cpus that can be elected have already the appropriate mapping.


BTW : Just in order to justify my request for misc. mitigations info in OPs comments, I wanted to first make sure that all of your cores were fully operational. They appear to be.

3
  • Sorry, I think I mislead you with the efficiency plot. That one is for the 4 cores CPU. So it is normal that I don't see efficiency improvements for -j valus greater than 5. Because I reached the maximum number of cores and the work is mainly CPU bound (because of aggressive IO caching) Jan 18 at 1:42
  • For the rescheduling part, I don't think that's the case. Because that wouldn't happen consistently throughout all test iterations. I also verify whether there are values that appear to be inconsistent and those are fortunately rare and shouldn't affect the median Jan 18 at 1:44
  • But you are right, the sudden increase in page fault at -j<ncores/2> is also worth investigating. But also outside the scope of this post. Jan 18 at 1:52
1

When make launch a process, the memory of the process is mapped to its files (executable, libraries). Then it is expected to see page faults. The more efficient the system is, the more page faults we can expect. It could be interesting to compare page fault rate with the efficiency (inverse of the time spent by make).

We can also have page faults when the system has not enough RAM and exchange data from/to the HDD. Here it is not a sign of efficency.

6
  • Hi, thanks for your answer :) I can assure you that RAM is not the issue. I have far more enough memory on my system than it is needed. And, the PC is not doing anything else other than repetitively compiling glibc. As for the bit with efficiency, that would make sense if I stopped the experiment at -jnproc. But when I go beyond that, the efficiency stays the same while the number of major page faults goes down. I added more info in the question Jan 14 at 23:40
  • 1
    Then the more process per seconds are launched, the more page faults are needed to load effectively the program in the process. You seem to have 4 cores, then if you make all of them busy with 4 processes you have the maximum of efficiency. We could have excepted a maximum at 5 (if the processes spend a little time waiting I/O). Jan 15 at 10:28
  • 1
    @FrédéricLoyer : Correct regarding the expectation of max efficiency at -j<Ncores+1> but it always appeared to me more because of the scheduler than because of the tasks which, from some point are essentially CPU bound. On a Core 2 no hyperthreding 10 years ago, -j 3 was indeed more efficient than -j 2 under CFS but this was not true under con kolivas' BFS.
    – MC68020
    Jan 15 at 12:35
  • @FrédéricLoyer I believe the reason why I do not see maximum efficiency at -j<Ncores+1> is because both my systems are aggressive on caching. Does this hypothesis make sense to you? Maybe because of the way they are configured, and have a lot of free RAM. 5000 and 500 are the values I found in /sys/block/sda/queue/iosched/write_expire and /sys/block/sda/queue/iosched/read_expire respectively. I should also mention that both systems use full drive encryption via lucks Jan 15 at 21:37
  • 1
    The more caching you have, the less IO you gave and the less incentive you have to add 1 (or more) to the number of CPU threading. Jan 15 at 21:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.