2

Our server solution handles many tunnel operation requests (add/update/delete) using the ioctl interface, and the AF_INET address family socket. Aprox. numbers are 100 operations/minute with over 8k tunnels total on the server.

We have observed a strange behavior with an increasing number of tunnels - as the number of tunnel grows, randomness of time spent in the kernel by ioctl rises.

Example: for about 100 tunnels, avg. ioctl operation time ranges from 100 to 150 ms, but with over 2000 it ranges from 100 to 10000 ms. The randomness of the process is surprising.

strace shows that over 97% of our process time is spent in the ioctl, and out internal timers confirm this.

Are there any ways to optimize this? Maybe by tweaking kernel parameters (which ones)? How to explain the randomness?

Our OS is Ubuntu Server 12.04.

1 Answer 1

1

It turned out that ioctl wasn't guilty here. strace showed it was, but a deeper analysis of the kernel with ftrace (or trace-cmd) showed that during ioctl processor scheduler caused most of the slow down during core/context switching.

2
  • so, how did you overcome this problem?
    – brauliobo
    Aug 27, 2015 at 0:50
  • 1
    renice critical processes so they get more and better processor time (helps a little), 'taskset' critical processes to fixed processor cores to minimize context switching (helps some more), set the affinity of your interfaces queues to all cores/processors (helps a lot!) (echo ffffffff > /sys/class/net/$interface/queues/rx-$i/rps_cpus and echo ffffffff > /sys/class/net/$interface/queues/tx-$i/rps_cpus)
    – er453r
    Sep 1, 2015 at 13:34

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .