3

I am using whisper.cpp to transcribe some sound files. It is a very CPU heavy process so I try to find some optimal settings and therefore I have done some tests with the thread setting (-t) but the results are super confusing. This is the command I execute:

date; time ./main -t [number of threads] -m ggml-model.bin -f 5min-16kHz.wav; date

I run this on a Macbook Pro with an Intel i7 with 6 cores (+ 6 hyperthread cores).

I have tried with the default settings (4 threads), 6 and 12 threads (and 14 threads but that didn't produce any output although all CPUs run at 100 %). Here is the result:

Threads output from time
4 1750.84s user 11.02s system 564% cpu 5:11.87 total
4 1862.04s user 18.63s system 553% cpu 5:39.58 total
6 2199.42s user 16.79s system 720% cpu 5:07.51 total
6 2212.72s user 14.49s system 722% cpu 5:08.22 total
12 4595.03s user 22.21s system 1053% cpu 7:18.47 total
12 4298.11s user 22.53s system 1059% cpu 6:47.85 total

As you can see, the CPU load increases as I increase the number of threads. You would expect the real time to decrease proportionally to the increase in CPU load (100 % for a minute should, approximately, correspond to 200 % for half a minute and 50 % for two minutes) but that doesn't happen here.

Instead I get approximately the same real time results with 4 and 6 threads while the CPU usage time increases with ≈ 25 % when running 6 threads. And 12 threads are even worse, the CPU-time doubles compared to 6 threads and the real time increases with 40 %.

I don't understand this. Of course, more threads don't scale linearly but CPU time should remain quite constant when performing the same task, independently of how many threads, shouldn't it? And real time should decrease when CPU-usage increases?

And considering the task and my hardware, what should be a reasonable setting for the number of threads to use? I was expecting it to be the number of cores + a little extra in case a thread waits for I/O. The sound file I process is 10 MB, whisper.cpp uses ≈ 3,6 GB on a computer with 32 GB (currently about 10 GB unused, memory pressure is "green").


Edit: corresponding values using only one thread (-t 1):

1619.90s user 20.86s system 197% cpu 13:48.78 total

Note that one thread used almost 200 % CPU. Not sure I understand that. But 13 minutes real time makes sense.

Edit 2: adding more CPUs (-p) made the performance worse.

-t 6 -p 3 - 6804.14s user 38.58s system 1040% cpu 10:57.84 total (twice as much real time, 3-4 times more CPU-time) -t 8 -p 2 - 10573.58s user 57.47s system 1018% cpu 17:23.63 total (more than 3 times as much real time and 6 time as much CPU-time) -t 4 -p 2 - 2962.38s user 28.65s system 854% cpu 5:50.01 total (approximately the same as with -t 4)

I think -p only should be used if you want to limit how much this task affects the computer. Otherwise, it will just use as many processers as it can.

I don't think it is I/O. It reads 3,08 GB in the first 5-10 seconds and then less than 10 MB for the rest of the run (that lasts at least 5 minutes).

Edit 3: using -t 13, that is, one more thread than my CPU supports, generates very odd results: 93213.70s user 450.23s system 978% cpu 2:39:36.88 total No, I am not joking, more than 50x as much CPU-time as -t 4, while CPU-usage is almost twice as high (978 % vs 564 %) and real-time increased more than 30x.

If I compare with -t 12 CPU-time increased by more than 20x, CPU usage is approximately the same, and real-time also increased by more than 20x. By just adding ONE more thread.

Something is iffy here, isn't it?

Edit 4:

Selected benchmark data

./bench -m ./models/ggml-small.en.bin -t 4

system_info: n_threads = 4 / 12 | AVX = 1 | AVX2 = 1 | AVX512 = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | VSX = 0 | 

whisper_print_timings:     load time =   540.82 ms
whisper_print_timings:   encode time =  3490.52 ms
whisper_print_timings:    total time =  4031.40 ms

5 threads are ≈ 8 % faster than 4 threads:

whisper_print_timings:     load time =   547.27 ms
whisper_print_timings:   encode time =  3193.27 ms
whisper_print_timings:    total time =  3740.58 ms

6 threads are 1 % slower than 5 threads:

whisper_print_timings:     load time =   591.16 ms
whisper_print_timings:   encode time =  3158.88 ms
whisper_print_timings:    total time =  3750.10 ms

7 threads are 15 % slower than 6 threads. And it is downhill from there. I guess this task only uses the 6 "real" cores I have, not the hyperthreading cores. I theory I guess 6 threads should be faster than 5 but I guess the computer performs some other tasks that interrupts one of the threads and uses one core from time to time when running this benchmark.

Edit 5:

Running the benchmark with a -20 nice value gave some interesting results (just listing the total time here)

Threads    Total time (ms)    ∆ (negative is better)
      4               3512    -13%
      5               3510     -6%
      6               3251 !! -13%
      7               3962     -8%

∆ is compared with the same number of threads with normal priority. 6 threads with high priority is 19 % faster than the default settings with normal priority.

3
  • I apologize, I realize my knowledge of your arch is far insufficient to understand what is going on in your 2 edits.
    – MC68020
    Commented Feb 10, 2023 at 19:59
  • @MC68020 I suspect something fishy is going on in the program. If I set the number of threads above 12 (but don't specify -p), all CPU-cores runs at 100 % but nothing happens. The same happens if I set -p to 4 - 100 % CPU but no result.
    – d-b
    Commented Feb 10, 2023 at 20:06
  • @MC68020 Furthermore, this weird correlation between CPU percentage and real time in the table - I suspect something in the code is weird.
    – d-b
    Commented Feb 10, 2023 at 20:07

2 Answers 2

1

Increasing the number of threads at some point makes the computation memory bound, so you stop gaining performance improvement. The CPU usage keeps increasing because in whisper.cpp we use busy loops to synchronise threads. This is much more CPU intensive and wastes a lot of CPU cycles, but helps reduce the latency by avoiding context switches and other side effects of mutex/condition variables.

The --processors parameter does not mean CPU processors. These are independent whisper.cpp processors that operate on the audio in parallel. You can read more about this functionality here:

https://github.com/ggerganov/whisper.cpp/pull/110

2
  • But what is the optimal number of threads? In all my tests I was far from running out of memory (I have 32 GB). How can I get closer to 1200 % "efficient" CPU-usage? Why do you want to reduce latency? This not a real time task/online - latency shouldn't matter, should it? I am happy to trade higher latency for higher overall performance!
    – d-b
    Commented Feb 14, 2023 at 23:12
  • Re --processors, wouldn't it be better overall (performance-wise) to work on multiple files in parallel (so you don't lose context) rather than work on a single file from multiple startpoints?
    – d-b
    Commented Feb 14, 2023 at 23:14
0

I have done some tests with the thread setting (-t)

That is indeed a pretty good idea… provided you also specify something for the -p option :

-p N, --processors N [1 ] number of processors to use during computation

Otherwise the number of processors used will default to 1. Leading your task to waste significant time context switching with no real profit. (the only profit could be to use the time one thread waits for IO to the profit of another runnable thread, but, acknowledging your app vastly CPU-bound...)

In fact, whatever the vastly CPU-bound application, with a single processor working, the higher the number of threads, the higher the waste… you just… proved it. ;-)

I would therefore suggest to :

  • 1/ Determine the basic performance with defaults t=4 ; p=1
  • 2/ t=4 and p=2 and increase t until insignificant benefit,
  • 3/ repeat 2 with higher values for p until… you start compromising other applications running.

Keeping in mind that I am convinced that devs are not prone to providing unwise default values, values that would condemned the performance of their app.
Applied here, it seems the dev acknowledges this app being quite significantly IO-bound and therefore advises a 4 to 1 t/p ratio.


UPDATE FOLLOWING OP'S EDITS

Disclaimer : Understanding Intel's Turbo Boost technology, in particular how the core i7 decides internally to modify core's frequency / offline cores and consequently its impacts on the performances of multi-threaded apps is beyond me.

EDIT 5 : This set of data does indeed provide logical figures :
Decreasing the nice value of a cpu-bound thread will increase the time it is allowed to keep the CPU when scheduled in consequently having the workload processed with significantly less costly context switches.
The set of data also suggests that only 6 cores are used. For what reason ? Turbo boost ? HT disabled ? If you get means to know precisely which ones of them…

8
  • I only have one CPU, but it has 6+6 cores? Or am I misunderstanding something? I will update the question with the results from running with just one thread.
    – d-b
    Commented Feb 10, 2023 at 18:35
  • Have added a few more datapoints. This is very odd. Any ideas what is going on?
    – d-b
    Commented Feb 11, 2023 at 0:08
  • 1
    @d-b : please run *bench" ( github.com/ggerganov/whisper.cpp/tree/master/examples/bench ) and compare your results with those you might find here github.com/ggerganov/whisper.cpp/issues/89 under a similar system.
    – MC68020
    Commented Feb 11, 2023 at 9:05
  • 1
    Added benchmark data.
    – d-b
    Commented Feb 11, 2023 at 17:25
  • @d-b : Updated my answer.
    – MC68020
    Commented Feb 12, 2023 at 9:19

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .