I'm running a test program which generates a large number of threads and asynchronous I/O. I'm seeing very high counts of these interrupts in /proc/interrupts, the program cannot scale beyond a certain point because CPUs are 100% saturated with softirq processing.

According to: http://www.kernel.org/doc/man-pages/online/pages/man5/proc.5.html CAL stands for 'remote function call interrupt' but that's all the info I can find on google. So...what does that mean? I have smp_affinity assigned for my I/O adapter cards and those are not the CPUs taking a huge number of CAL interrupts. Is it a sign my program is running with incorrect SMP affinity?

  • Without seeing the source code of the program, or at least a simplified pseudocode example, it is difficult to say what is wrong. Also what do you mean by "asynchronous IO"? If you mean posix aio, then the whole point is to be able to queue a bunch of background IO without having to create a bunch of threads. The more threads you create, the more overhead there will be, so how many threads are we talking about here?
    – psusi
    Apr 30 '12 at 23:37
  • I'm using a recent version of fio [freecode.com/projects/fio], configured to run ~20 "jobs" at the same time, using libaio to keep multiple random I/Os queued up to each drive. To be honest I'm not sure how many threads it's spawning. I did notice that when I disabled my second CPU socket (leaving only socket 1 enabled), I got similar I/O performance from fio but the number of CAL interrupts dropped dramatically. So maybe it's due to fio threads running on socket 2, invoking interrupt handlers that are affinitized to socket 1? I'm wondering if I can do some kind of trace or profiling.
    – twblamer
    May 1 '12 at 20:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.