0

I have a job on a managed Kubernetes cluster that is a bit IO intensive. Nothing crazy, it downloads about 1gb of data, writes it on /tmp folders, does some minor edits, compress and encrypts it, and uploads it again somewhere else. It runs every 2 hours.

Since a few days our cloud provider support, which is a big one with a name starting by a A and ending with a E, is blaming all the Kubernetes issues we experience on this job, because it creates a lot of load. I think it's a bit crazy that such a job could cause any issue in Kubernetes and I think the support is not very good but they may be right.

So I am looking for solutions to reduce the IO load, the job is IO bound. As almost everything running on this cloud provider by the way, they offer really poor IO performances unless you throw real money at them.

It seems that Kubernetes doesn't support limits on IOs yet. So one obvious solution would be to move the job out of Kubernetes, even out of this cloud, but it would start a process and soon I could see myself running everything on an old laptop next to the coffee machine, with better performances and availability.

As an alternative, I was thinking about throttling calls to read and write from the libc using LD_PRELOAD. To inject some nanosleep calls if a speed higher than what can be offered by an old mass storage USB-2 stick is detected.

Do you think this could work ? Has someone done that before ?

Improve this question
4
  • 1
    You can give the job a dedicated volume so it's IO will not impact the IO of other pods: kubernetes.io/docs/concepts/storage/volumes. if you also need to isolate it on a network level, the easiest route might be putting it on its own node via a selector.
    – jordanm
    Jul 2, 2021 at 20:54
  • Thanks for the suggestion. Actually, it's already on a dedicated volume and on a specific node with little load.
    – e741af0d41bc74bf854041f1fbdbf
    Jul 3, 2021 at 9:54
  • 2
    It's probably possible, but it's not a good idea. Keep in mind that a LD_PRELOAD lib which catches the write(2) function will not catch fwrite(3) or printf(3) -- no matter what you read on sites like this ;-)). How about using ionice(1) instead? (not tested).
    – user313992
    Jul 3, 2021 at 9:54
  • 1
    It's probably related to "Kubernetes doesn't support limits on IOs yet", but cgroups v2 (rather than only cgroups v1) is a method to provide I/O limit. So Kubernetes (and probably Docker) must have complete support for cgroup v2 to offer this feature. From what I understood that's because it involves interactions with memory (cache) and blocks and v1 considers them separately. There's a good blog on limiting I/O with cgroups there: andrestc.com/post/cgroups-io
    – A.B
    Jul 3, 2021 at 11:10

0

Reset to default

You must log in to answer this question.

Browse other questions tagged .