0

Newbie systemd user here; my apologies if the question seems 'too basic' and would be better asked elsewhere on the StackOverflow ecosystem...

I'm in the process of converting some of my ancient init service files, written 12 years ago or so, to systemd. Some of those had rather nasty tricks to figure out how to configure a batch of services; I'm not really an experienced system programmer, and I'm fond of quick & dirty hacks to get things working as proofs-of-concept... they can always be improved later on.

That said, here is what I'm trying to accomplish: I'm hosting an online virtual world platform (no, it's not Minecraft...) that runs several instances of the same binary, but with different parameters (think shards). This would be rather easy to do if I knew how many instances to launch in advance (i.e. at boot time), or at least what set of parameters they would take, or even if I had an actual list (potentially generated in advance) with that information.

In practice, what happens is the following: there is a special 'configuration' directory (at least that one is known in advance!) which has several subdirectories (the number is not known in advance!), each named after a specific instance. The name is important (i.e. I cannot make things easier by using sequential names, which would make some tricks easier to apply, even assuming that I knew in advance how many subdirectories there are — which I don't!). Imagine that the subdirectory names are city names, for example, NewYork, Chicago, LA, KansasCity.

So, schematically, you could think of the overall directory tree as looking like this:

game-root-folder 
|
\___ bin
|     \___________ myapp (the actual game engine for each instance)
|     \___________ myapp-single-instance.sh (explained below)
|
\___ config-folder
       \__________ NewYork
       |              \______ config.ini
       |          
       \__________ Chicago
       |              \______ config.ini
       |          
       \__________ LA
       |              \______ config.ini
       |          
       \__________ KansasCity
       :              \______ config.ini

(my apologies, I'm not good with ASCII diagrams...)

Assume that each config.ini is just a set of GPS coordinates for the city in question (in practice, it's not much more complex than that, but what matters is that there are enough items in those files — which might be not written by a human, but generated automatically from a 'managing app' — to make it very hard to encode everything in environmental variables), i.e. something which is unique and different from city to city and which is hard-coded into the startup configuration for that city.

In practice, the actual layout is considerably more complex, but this should be enough to explain my issue.

When not using systemd, but rather launching everything manually from the shell, the task is simple — the command to launch each instance will look something like this:

# Call this script with `myapp-single-instance.sh start <city name>`
case "$1" in
    start)
        cd /full/path/to/game-root-folder/bin
        /usr/bin/screen -S $2 -d -m -l myapp \
          --config=/path/to/game-root-folder/config-folder/$2/config.ini
        ;;
    stop)
        # discussed below
        ;;
    *)
        # show usage
        exit 1
        ;;
esac
        

The non-systemd configuration file just had an additional script that would list the contents of that configuration directory and extract the names of the subdirectories, launching the start/stop script for each city name, i.e. something like this:

cd /full/path/to/game-root-folder/bin
for CITY_NAME in `ls /path/to/game-root-folder/config-folder`
do
    myapp-single-instance.sh start $CITY_NAME
done

As a side-note, to stop those instances, one can do:

/usr/bin/screen -S $CITY_NAME -X eval 'stuff "quit"\015'

(assuming that quit is the console command to close that instance) or, if the above doesn't work for some reason (instance in endless loop, not accepting commands):

/usr/bin/screen -X -S $CITY_NAME kill

in which case kill is a command of screen itself, to gracefully terminate itself and whatever is inside it. The last option is sending a SIGKILL, of course (also done via the currently existing scripts).

As you can see, if someone wants to add a new instance for a new city, all they need to do is to create a new subdirectory, say, Boston, and start it with the single-line start/stop script; individual instances can then be stopped or started manually if needed; new instances might require manual intervention anyway (putting those config.ini in place), so it's okay to expect that there are additional commands to set up a new instance — so long as each instance is mostly independent of the others, and does not interact directly with them, there is no need to signal anything to the existing instances when adding a new one (or removing an existing one). Again, in reality, there is a 'master' instance monitoring server, which does some housekeeping behind the scene, but for the purpose of this question, let's ignore its existence for now.

The above example is trivial to adapt to a init scenario.

Now, how to do it under systemd? Well, the two-tiered approach seems to fit like a glove to the usage of systemd targets, launching several instances from a template. Here is my naïve attempt, loosely based on this answer (point 3.), but also on others:

; [email protected]

[Unit]
Description=Game city name %I
After=syslog.target
After=network.target
Requires=mariadb.service mysqld.service
PartOf=myapp.target

[Service]
Type=forking
User=myappuser
Group=myappgroup
WorkingDirectory=/full/path/to/game-root-folder/bin
ExecStart=myapp-single-instance.sh start "%I"
ExecStop=myapp-single-instance.sh stop "%I"
ExecStop=/bin/sleep 5
KillSignal=SIGCONT
Restart=always
RestartSec=30s
Environment=USER=myappuser HOME=/full/path/to/game-root-folder/bin
RemainAfterExit=false
SuccessExitStatus=1

[Install]
WantedBy=multi-user.target

So far, so good, I'm basically just encapsulating my existing scripts inside the systemd configuration.

But now what to do with the target?

; myapp.target
[Unit]
Description=Launch all game instances
[email protected] [email protected] [email protected]

[Install]
WantedBy=multi-user.target

Whoops, I forgot Chicago! See, that's what happens when manually editing configurations...

Basically, I would like to have a dynamic way to do that, just like I did with the amazing `ls /path/to/game-root-folder/config-folder` command — so easy, but so powerful! However, from what I read on the subject, it's not possible to pipe the output of a shell command to the Wants=... line — in fact, the only lines that can run shell commands are the Exec...= ones (for obvious reasons). Tough luck for me!

Others have struggled with similar issues. The solution was to push the logic of selecting what instance to run to environment variables, which could then be 'massaged' and sent to the executable itself. I'm a bit baffled about how such a solution could even be implemented in my case, but it doesn't seem to fit my requirements. In fact, I find it strange that such an 'obvious' use-case — dynamically calling service units from a target (where 'dynamic', in this context, means 'a set of service units for a single executable, of unknown size, where each element may have a different range of parameters, different from each other, and not known in advance') — is not a 'trivial' scenario; perhaps I'm not fully grasping how systemd addresses this case?

I have seen two examples where the solution was to include different files (using the systemd override mechanism, i.e. putting files inside /etc/systemd/myapp.service.d/), and get systemd to search for those additional configurations; or, alternatively, list all the possible instances to be called from the .target script, store them either in a file or even into an environment variable. Every time someone adds an additional instance, a background-timed process would check the subdirectories for each instance and either return an environment variable with every bit of collected data, or write that to a special file (one would assume that this file would be shared on a well-known spot), and read from it in order to figure out how to start each instance at boot time. Such solutions imply running a background daemon that checks what files are available and refreshes such file(s) before calling the target unit with a reload. Nevertheless, these solutions seem not to work when trying to change the Wants=.. or Require=.. lines; it's mostly the Exec...= parameters that can be addressed this way (although I saw a few conflicting instructions, where allegedly some systemd versions have code that will have a 'dynamic' way of filling in fields...).

Also, note that I would like to retain the ability to start/stop/reload instances individually, as well as get them all started at boot time (or gracefully stopped before a mandatory reboot, for instance).

Taking all the above into account, what solution would you recommend?

Thanks in advance and keep up the awesome work in this community!

Cheers,

  • Gwyn
10
  • 1
    You might look at man systemd.path for path activation, i.e. calling a script when a new file appears. It probably just uses inotify.
    – meuh
    Oct 20, 2021 at 11:23
  • Aye, @meuh, I thought that I could 'cook' something with that: the idea would be to launch something that did my magical ` ls ... ` and wrote the results... somewhere (a file? env?). However, I'd be still stumped: how to get the contents of that into the Wants=... line? Unless, of course, you're suggesting auto-modifying code... eek! 😅 Oct 20, 2021 at 20:36
  • 1
    As I understand it, a Wants= in myapp.target is the same as a WantedBy=myapp.target in [email protected]. They just create a symbolic link in myapp.target.wants/ when you do systemctl enable myapp@xxx, and I'm not sure you are even doing that.
    – meuh
    Oct 21, 2021 at 8:58
  • 1
    If PartOf is part of the service [Unit] then the target shouldn't need Wants; systemd will automatically know what templates have been enabled for the target. systemctl enable myapp@Chicago (etc) should work with what you have. If you remove Wants= from the target and systemctl enable myapp and then systemctl start myapp what happens? Nov 4, 2021 at 21:54
  • 1
    Well, well, so that's how the magic is done... @JosephTingiris, removing the Wants= line from the target most certainly works, exactly as intended! After your explanation (and, to a degree, @meuh's) it kind of makes sense, too: after all, that's exactly the use case which I would think was commonplace! Nov 6, 2021 at 0:31

1 Answer 1

1

If PartOf is part of the service [Unit] then the target shouldn't need Wants; systemd will automatically know what templates have been enabled for the target.

systemctl enable myapp@Chicago (etc) should work with what's in the descrption.

Remove Wants= from the target and systemctl enable myapp and then systemctl start myapp ... what happens?

1
  • 1
    "Answers" that contain questions like "what happens if ...?" should be comments, not "answers" IMHO.
    – U. Windl
    Nov 24, 2023 at 10:20

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .