When using cgroups (either standalone or as part of docker or kubernetes)
memory.limit_in_bytes counts file cache towards the processes memory usage. This has an unexpected side effect that I'm trying to avoid.
Normally, when you run a process outside a cgroup, if you issue many writes, those tend to accumulate into the block cache, once the cache fills, it will stall IO operations until there's enough room to keep writing.
If you're running inside a cgroup (e.g. docker container) that has a memory limit and you issue many writes fast, the behavior when you reach the memory limit due to outstanding writes is unpredictable.
If you get there, but the host's block cache has room, IO operations rather than stall, fail with ENOBUF or similar error, this causes many applications to fail, on the other hand if the block cache is full, you get a stall as usual.
Is there any way to make
memory.limit_in_bytes ignore the file cache or at least avoid the inconsistent behavior without modifying said applications.