26

Is there a way to dynamically assign environment variables in a systemd service unit file?

We have a machine that has 4 GPUs, and we want to spin up multiple instances of a certain service per GPU. E.g.:

  • gpu_service@1:1.service
  • gpu_service@2:1.service
  • gpu_service@3:1.service
  • gpu_service@4:1.service
  • gpu_service@1:2.service
  • gpu_service@2:2.service
  • gpu_service@3:2.service
  • gpu_service@4:2.service
  • ad nauseam

So the 1:1, 2:1, etc. are effectively the %i in the service unit file.

In order for the service to bind to a particular GPU, the service executable checks a certain environment variable, e.g.:

USE_GPU=4

Is there a way I can take %i inside the service unit file and run it through some (shell) function to derive the GPU number, and then I can set the USE_GPU environment variable accordingly?

Most importantly, I don't want the hassle of writing multiple /etc/systemd/system/gpu_service@x:y.service/local.conf files just so I can spin up more instances.

5 Answers 5

23

If you are careful you can incorporate a small bash script sequence as your exec command in the instance service file. Eg

ExecStart=/bin/bash -c 'v=%i; USE_GPU=$${v%:*} exec /bin/mycommand'

The $$ in the string will become a single $ in the result passed to bash, but more importantly will stop ${...} from being interpolated by systemd. (Earlier versions of systemd did not document the use of $$, so I don't know if it was supported then).

4
  • 1
    I ended up doing something like that. :)
    – Kal
    Nov 18, 2016 at 8:10
  • 10
    Call a bash -c to start a program from the unit file? Call exec? This is like using a forklift on top of a forklift (maybe with another forklift on top) because the first forklift has trouble actually forklifting. Dec 31, 2018 at 18:50
  • Something like this would work. Or a wrapper script to split that variable :) The other bizarre option would be to create another service to "run beforehand" and setup an EnvironmentFile, not sure how that would work with templates tho: stackoverflow.com/a/42841480/32453
    – rogerdpack
    Jul 6, 2020 at 15:18
  • 2
    @DavidTonhofer How would a solution without "double forklifter" look like? I see that the bash is there to setup and mangle %i into USE_GPU. Then exec replaces the shell process with the real one.
    – U. Windl
    Feb 10, 2021 at 13:34
10

No built in way. You need to do these things before your service starts. One way would be putting it to an environment file.

[Service]
# Note you need to escape percentage sign
ExecStartPre=/bin/sh -c "my_awesome_parser %%i > /run/gpu_service_%i"
EnvironmentFile=/run/gpu_service_%i
ExecStart=...
1
  • 5
    Unfortunately you can't use an ExecStartPre to write an env file, then use it, apparently it has to be written beforehand.
    – rogerdpack
    Jul 6, 2020 at 15:15
4

It looks like you can indeed set environment variables inside a systemd unit file...

Per suggestions from commenters, here is the solution:

Using environment variables in systemd units

Environment directive

systemd has an Environment directive which sets environment variables for executed processes. It takes a space-separated list of variable assignments. This option may be specified more than once in which case all listed variables will be set. If the same variable is set twice, the later setting will override the earlier setting. If the empty string is assigned to this option, the list of environment variables is reset, all prior assignments have no effect. Environments directives are used in built-in Container Linux systemd units, for example in etcd2 and flannel.

With the example below, you can configure your etcd2 daemon to use encryption. Just create /etc/systemd/system/etcd2.service.d/30-certificates.conf drop-in for etcd2.service:

[Service]
# Client Env Vars
Environment=ETCD_CA_FILE=/path/to/CA.pem
Environment=ETCD_CERT_FILE=/path/to/server.crt
Environment=ETCD_KEY_FILE=/path/to/server.key
# Peer Env Vars
Environment=ETCD_PEER_CA_FILE=/path/to/CA.pem
Environment=ETCD_PEER_CERT_FILE=/path/to/peers.crt
Environment=ETCD_PEER_KEY_FILE=/path/to/peers.key

Then run sudo systemctl daemon-reload and sudo systemctl restart etcd2.service to apply new environments to etcd2 daemon.

Quoted text taken from the following URL: https://coreos.com/os/docs/latest/using-environment-variables-in-systemd-units.html

5
  • 3
    While this may theoretically answer the question, it would be preferable to include the essential parts of the answer here, and provide the link for reference. Feb 23, 2017 at 5:12
  • 3
    While your comment may theoretically improve my future responses in stackexchange, it would be preferable for you to have included the essential parts of the answer in your comment instead of just commenting to point out how incompetent someone may be :)
    – CyberK
    Mar 1, 2017 at 3:43
  • 2
    Welcome to Stack Exchange! Thanks for the comment, you made me smile. Also thanks for taking the time to edit your answer. We are trying to build something that will have value over time, and link only answers just don't age very well. Mar 2, 2017 at 8:06
  • If you add Environment=ABC=%i it does set that env. variable "to the entirety of %i". I guess you could make a wrapper to strip off the "stuff beyond the quote" you don't want, and it call the real executable. But if you're making a wrapper, you could even just pass %i in as an argument to it ex: ExecStart=my_wrapper %i
    – rogerdpack
    May 28, 2019 at 20:02
  • 5
    The question was for "dynamic" variables; you've simply given us the answer for the static solution.
    – Otheus
    Feb 12, 2020 at 11:38
1

It's ugly and not quite what you asked for, nor does it allow for autostart, but for followers it is possible to do something using the systemctl environment:

$ sudo systemctl set-environment USE_GPU=4 # add it to the env. variables for future services
$ sudo systemctl start gpu_service@4:2.service

One problem is if you run/start more than one "at once" (in parallel) then it could cause confusion/race conditions.

Just trying to list all the options possible, some of the other answers work as well :)

1
  • 4
    +1: ExecStartPre=/bin/bash -c "/bin/systemctl set-environment USE_GPU=4" is the best way to do this of the answers posted here.
    – ZachB
    Jul 3, 2020 at 18:56
0

Use the EnvironmentFile directive which points to a env file with the variable. The file is created by an ExecStartPre command before the actual service is started. Your variable will be accessible in the ExecStart directive with ${GPU_COUNT}.

[Unit]
Description=Dynamic variables in systemd units

[Service]
Type=simple
EnvironmentFile=-/tmp/gpu.env
ExecStartPre=sh -c 'printf "%%s\n" GPU_COUNT=$( (...) ) > /tmp/gpu.env'
ExecStart=(...)
ExecStartPost=sh -c 'rm /tmp/gpu.env'

You have to prefix the file path with -, which makes systemd ignore the fact that the file doesn't exist during the initial check. Later, the ExecStartPre command creates the env file. In another unit state the file location will be evaluated again (still before ExecStart), and if it exists, systemd will load the variables. Here's how man systemd.exec explains it:
The files listed with this directive will be read shortly before the process is executed (more specifically, after all processes from a previous unit state terminated. This means you can generate these files in one unit state, and read it with this option in the next. The files are read from the file system of the service manager, before any file system changes like bind mounts take place).

2
  • Are you sure about this? On my embedded system with systemd 251.8 EnvironmentFile is evaluated only before ExedStartPre and not repeated before ExecStart. Thus the env file is not read into env of ExecStart.
    – Martin H.
    Feb 7 at 15:05
  • 1
    @Martin H. I have been running a few systemd services using this approach without issues, on Ubuntu x86 machines, with systemd 252.x at the time of writing the answer. The earliest accessible man page on freedesktop, which includes the quoted sentence, comes from systemd 247.
    – BMC
    Feb 8 at 20:48

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .