Dynamic variables in systemd service unit files

Question

Is there a way to dynamically assign environment variables in a systemd service unit file?

We have a machine that has 4 GPUs, and we want to spin up multiple instances of a certain service per GPU. E.g.:

• gpu_service@1:1.service
• gpu_service@2:1.service
• gpu_service@3:1.service
• gpu_service@4:1.service
• gpu_service@1:2.service
• gpu_service@2:2.service
• gpu_service@3:2.service
• gpu_service@4:2.service
• ad nauseam

So the 1:1, 2:1, etc. are effectively the %i in the service unit file.

In order for the service to bind to a particular GPU, the service executable checks a certain environment variable, e.g.:

USE_GPU=4

Is there a way I can take %i inside the service unit file and run it through some (shell) function to derive the GPU number, and then I can set the USE_GPU environment variable accordingly?

Most importantly, I don't want the hassle of writing multiple /etc/systemd/system/gpu_service@x:y.service/local.conf files just so I can spin up more instances.

Answer 1

If you are careful you can incorporate a small bash script sequence as your exec command in the instance service file. Eg

ExecStart=/bin/bash -c 'v=%i; USE_GPU=$${v%:*} exec /bin/mycommand'

The $$ in the string will become a single $ in the result passed to bash, but more importantly will stop ${...} from being interpolated by systemd. (Earlier versions of systemd did not document the use of $$, so I don't know if it was supported then).

Answer 2

No built in way. You need to do these things before your service starts. One way would be putting it to an environment file.

[Service]
# Note you need to escape percentage sign
ExecStartPre=/bin/sh -c "my_awesome_parser %%i > /run/gpu_service_%i"
EnvironmentFile=/run/gpu_service_%i
ExecStart=...

Answer 3

It looks like you can indeed set environment variables inside a systemd unit file...

Per suggestions from commenters, here is the solution:

Using environment variables in systemd units

Environment directive

systemd has an Environment directive which sets environment variables for executed processes. It takes a space-separated list of variable assignments. This option may be specified more than once in which case all listed variables will be set. If the same variable is set twice, the later setting will override the earlier setting. If the empty string is assigned to this option, the list of environment variables is reset, all prior assignments have no effect. Environments directives are used in built-in Container Linux systemd units, for example in etcd2 and flannel.

With the example below, you can configure your etcd2 daemon to use encryption. Just create /etc/systemd/system/etcd2.service.d/30-certificates.conf drop-in for etcd2.service:

[Service]
# Client Env Vars
Environment=ETCD_CA_FILE=/path/to/CA.pem
Environment=ETCD_CERT_FILE=/path/to/server.crt
Environment=ETCD_KEY_FILE=/path/to/server.key
# Peer Env Vars
Environment=ETCD_PEER_CA_FILE=/path/to/CA.pem
Environment=ETCD_PEER_CERT_FILE=/path/to/peers.crt
Environment=ETCD_PEER_KEY_FILE=/path/to/peers.key

Then run sudo systemctl daemon-reload and sudo systemctl restart etcd2.service to apply new environments to etcd2 daemon.

Quoted text taken from the following URL: https://coreos.com/os/docs/latest/using-environment-variables-in-systemd-units.html

Answer 4

It's ugly and not quite what you asked for, nor does it allow for autostart, but for followers it is possible to do something using the systemctl environment:

$ sudo systemctl set-environment USE_GPU=4 # add it to the env. variables for future services
$ sudo systemctl start gpu_service@4:2.service

One problem is if you run/start more than one "at once" (in parallel) then it could cause confusion/race conditions.

Just trying to list all the options possible, some of the other answers work as well :)

Answer 5

Use the EnvironmentFile directive which points to a env file with the variable. The file is created by an ExecStartPre command before the actual service is started. Your variable will be accessible in the ExecStart directive with ${GPU_COUNT}.

[Unit]
Description=Dynamic variables in systemd units

[Service]
Type=simple
EnvironmentFile=-/tmp/gpu.env
ExecStartPre=sh -c 'printf "%%s\n" GPU_COUNT=$( (...) ) > /tmp/gpu.env'
ExecStart=(...)
ExecStartPost=sh -c 'rm /tmp/gpu.env'

You have to prefix the file path with -, which makes systemd ignore the fact that the file doesn't exist during the initial check. Later, the ExecStartPre command creates the env file. In another unit state the file location will be evaluated again (still before ExecStart), and if it exists, systemd will load the variables. Here's how man systemd.exec explains it:
The files listed with this directive will be read shortly before the process is executed (more specifically, after all processes from a previous unit state terminated. This means you can generate these files in one unit state, and read it with this option in the next. The files are read from the file system of the service manager, before any file system changes like bind mounts take place).