0

I have a Linux web server (Debian, just in case that made a difference), which is Apache + PHP-FPM. Each Apache VirtualHost uses a dedicated PHP-FPM pool (through a dedicated unix socket), which runs under a dedicated system user.

For example, let's say I have the example.com VirtualHost. Then, in the VirtualHost Apache config, I have something like

ProxyPassMatch ^/(.*\.php(/.*)?)$ unix:/run/php/php7.3-fpm-examplecom.sock|fcgi://127.0.0.1:9000/home/examplecom/htdocs/www.example.com/

And, in /etc/php/7.3/fpm/pool.d/ I have the corresponding configuration file for the PHP-FPM pool, namely examplecom.conf, which contains

[examplecom]
user = examplecom
listen = /run/php/php7.3-fpm-examplecom.sock
; and other things...

It happens, more often than not, some website gets hacked (forgot to update, installed shameless insecure plugins, you name it) and starts hogging available server resources: in my case hard disk I/O bandwidth is the most common.

In order to avoid DoS-ing all the websites on the same server, I'd like to limit the hard disk I/O bandwidth per website, so that a hacked website goes down alone and it does not take all the others with it.

How do I limit the hard disk I/O bandwidth per website? Well, I think I can limit that per user instead, since each VirtualHost has a dedicated user. And how do I do that? Well, Google suggested me cgroups.

However I couldn't find any example about how to add a user, instead of a process ID, to a control group.

Am I looking for the wrong thing? Or am I searching for the wrong keywords? I don't know, but the question is: how do I limit per user hard disk I/O bandwidth using cgroups under Linux?

  • The cgroups subsystem works on processes and threads. What you could do with cgroups is confining the processes/threads of a particular user to a particular cgroup. If your Apache+FPM configuration really uses PAM then you can do that through the PAM session configuration of your service, e.g. by using pam_cgroup or perhaps by using a pam_exec script that moves the processes into the cgroup they belong to. Then tune that cgroup's blkio (if you use cgroupsv1) or io (if you use cgroupsv2) controller. Dive into man cgroups. – LL3 Aug 17 at 15:04
  • You say "if it really uses PAM": that "really" makes me wonder if it is a strange thing. Doesn't the pool configuration I posted make PHP-FPM use PAM to switch to the specified user? – Lucio Crusca Aug 17 at 19:27
  • It's not strange at all, it's just that I know little about current Apache and nothing at all about PHP+FPM. To be honest I see nothing referring explicitly to PAM in that snippet but I trust you if you say it uses PAM under the hood. – LL3 Aug 17 at 21:21
  • No, please do not trust me, I'm not a reliable source in this case. I've used the "PAM user" expression as a synonym of "system user", but now I realize it was a bad idea. I'm going to edit my question and remove PAM references. – Lucio Crusca Aug 18 at 5:43
0

Thanks to LL3 comments to my question, I figured out I had to resort to a custom script. So here it is. It looks for the PHP-FPM processes, it takes respective running users and PIDs, it creates the cgroups and it inserts the PIDs in the correct cgroups.

Then I configured a crontab entry so that the script runs once per hour, which is more than enough in my case: Linux automatically puts child processes into the parent process cgroup, so this scripts actually needs to run only once just after every PHP-FPM restart (such as at boot time or when I add a new website/user). Running it once per hour it's useful to avoid forgetting to run it manually when it's due.

This script assumes the /cgroups directory was mounted with the cgroup2 filesystem, so you should arrange for it beforehand. You can add an entry into /etc/fstab and, if your kernel mounts cgroups version 1 by default, configure /etc/default/grub to disable v1 and reboot (hint: add cgroup_no_v1=all to GRUB_CMDLINE_LINUX_DEFAULT and then run update-grub && reboot).

It has been running in production for a month now and it seems to work. I'm pretty sure there are compromised websites at least in one of the two servers where I put this, and it does a good job in limiting the damages (unrelated sites go down only for a few seconds now and then).

Please note that, since the first version of this script, I had to edit it to make it monitor CPU and memory too, because monitoring disk I/O alone was not enough.

#!/bin/bash

DEVICE="254:0" # see /proc/partitions, this is /dev/vda in my case
WRITE_IOLIMIT=2097152 # 2MB/s, choose this as a fraction of your disk max write rate, I tested mine with `dd if=/dev/vda of=/testmaxspeed.dd bs=1M count=10000 status=progress`
READ_IOLIMIT=4194304 # 4MB/s, choose this as a fraction of your disk max read rate, I tested mine with `dd if=/dev/vda of=/dev/null bs=1M count=10000 status=progress`

function cleanup
{
  rm -f "$PLISTFILE" >/dev/null 2>&1
}

trap cleanup INT TERM EXIT

PLISTFILE=$(mktemp)

ps axo user:40,pid,comm | grep php-fpm | grep -v 'grep php-fpm' | grep -v '^root ' | tr -s ' ' | cut -d' ' -f 1-2 | sort | uniq > $PLISTFILE

cat $PLISTFILE | while read ROW ; do
  USERNAME=$(echo "$ROW" | cut -d' ' -f1)
  mkdir /cgroups/$USERNAME 2>/dev/null
  echo "$DEVICE rbps=$READ_IOLIMIT" > /cgroups/$USERNAME/io.max
  echo "$DEVICE wbps=$WRITE_IOLIMIT" > /cgroups/$USERNAME/io.max
  echo 402653184 > /cgroups/$USERNAME/memory.high # throttle when user is above 384MB of RAM
  echo 603979776 > /cgroups/$USERNAME/memory.max # OOM-Kill when user is above 576MB of RAM
done

echo "+io" > /cgroups/cgroup.subtree_control
echo "+cpu" > /cgroups/cgroup.subtree_control
echo "+memory" > /cgroups/cgroup.subtree_control

cat $PLISTFILE | while read ROW ; do
  USERNAME=$(echo "$ROW" | cut -d' ' -f1)
  PROCESSID=$(echo "$ROW" | cut -d' ' -f2)
  echo $PROCESSID > /cgroups/$USERNAME/cgroup.procs  
done

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.