Start a huge number of services with systemd

Question

I want to start a huge number (e.g. 256) of services with systemd. Fortunately, systemd offers slices which makes it pretty easy to reuse the service description. But starting many processes at once, kills the system, because every process does some start-up computation, which pushes the load above 50.

Is it either possible to define a dependency chain for all services with slices, After=service@%(i-1).service Wants=service@%(i-1).service? Or is it possible to define something like a service pool and systemd manages the start of the services, i.e. not running all at once but starting maybe 10 at once and after this the next block?

Answer 1

I found a nice solution using ExecStartPost.

% systemctl --user cat example@ 
# /home/joerg/.config/systemd/user/example@.service
[Unit]
Description=Example of service farm; instance %i

[Service]
Type=simple
ExecStart=/bin/sleep 99999
ExecStartPost=/bin/sh -c 'test %i -gt 0 || exit 0 ; systemctl --no-block --user start %p@$((%i - 1))'

% systemctl --user start example@2

% systemctl --user status example.slice
● example.slice
   Loaded: loaded
   Active: active since Thu 2017-04-27 11:04:43 CEST; 27min ago
   CGroup: /user.slice/user-1000.slice/user@1000.service/example.slice
           ├─example@0.service
           │ └─19423 /bin/sleep 99999
           ├─example@1.service
           │ └─19420 /bin/sleep 99999
           └─example@2.service
             └─19417 /bin/sleep 99999

Answer 2

No, systemd doesn't support a syntax like Wants=service@%(i-1).service?.

However, you could write a simple script that writes out 256 similar systemd unit files which contain an explicit dependency chain. Here are some other patterns to consider:

1. systemd allows you to set resource control parameters at run time, with syntax like:

   systemctl --runtime set-property foobar.service CPUShares=777

So, you could throttle your units during start up to each use less CPU, then once things have calmed down, allow them to use more CPU. This seems harder than need to be, which brings me to the next option...

2. In man systemd.resource-control, you'll find that there is a StartupCPUShares= option which is distinct from the CPUShares= option. I would experiment with throttling CPU with StartupCPUShares= to see if produces the result you want.

Personally, I used a very low-tech route to solve a problem like this. I started my series of services one at a time, with a "sleep" between service start. This used a little bash scripting instead of system, but has worked well enough. Thankfully, I don't also have the requirement to bring all the services online absolutely as soon as possible.

The next iteration of the system I will likely design around systemd and will like try using StartupCPUShares= myself to see if it's a better way to solve the problem.