0

I'm trying to set up the Hashicorp Vault Agent as a systemd service. I can manually run that agent with the user vault.

Note, perhaps that's important: here's the /etc/passwd for that user :

vault:x:994:989::/home/vault:/bin/false

So I need to do sudo su -s /bin/bash vault to get a vault session.

With that in mind, I can do the vault agent -config=<pathToConfig>and it works.

Now here the /usr/lib/systemd/system/vault-agent.service I've set up :

[Unit]
Description="HashiCorp Vault - A tool for managing secrets"
Documentation=https://www.vaultproject.io/docs/
Requires=network-online.target
After=network-online.target
ConditionFileNotEmpty=/etc/vault.d/vault.hcl

[Service]
User=vault
Group=vault
ProtectSystem=full
ProtectHome=read-only
PrivateTmp=yes
PrivateDevices=yes
SecureBits=keep-caps
AmbientCapabilities=CAP_IPC_LOCK
Capabilities=CAP_IPC_LOCK+ep
CapabilityBoundingSet=CAP_SYSLOG CAP_IPC_LOCK
NoNewPrivileges=yes
ExecStart=/bin/vault agent -non-interactive -config=/etc/vault.d/agent-config-prod.hcl
ExecReload=/bin/kill --signal HUP $MAINPID
KillMode=process
KillSignal=SIGINT
Restart=no
RestartSec=5
TimeoutStopSec=30
StartLimitIntervalSec=60
StartLimitBurst=3
LimitNOFILE=65536

[Install]
WantedBy=multi-user.target

This is a service conf I've found multiple times.

But I always get the same issue: Error storing PID: could not open pid file: open ./pidfile: permission denied

I tried to replace the ExecStart= by /bin/whoami, just to be sure, yes, it's indeed vault. Permission and location of that ./pidfile (default install location):

/etc/vault.d/pidfile

drwxr-xr-x. 108 root root  8192 May 15 16:32 etc
drwxr-xr-x   3 vault vault     113 May 15 17:43 vault.d
-rwxrwxrwx 1 vault vault   0 May 15 17:48 pidfile #not default permission, but I am desesperate.

I am really suspicous about the sudo su -s /bin/bash vault command that, perhaps, grants the vault user more privileges. If so, how to incorporate it into my service?

I ran systemctl reload daemon everytime and SELinux is disable.

ps: if someone has a great link about how to set up a systemd for the vault AGENT (not as root), I'll take it.

EDIT : about the sudo -s /bin/bash vault

$ sudo -s /bin/bash vault
/bin/vault: cannot execute binary file
$ su -s /bin/bash vault
Password: (and I have no password or I don't know it)

So that's why I'm using the full sudo su -s /bin/bash vault command.

Improve this question
6
  • Couple things, fyi. su is an older command that does the same job as sudo but slightly different. There is (should be) no benefit to running them both together. If you want a root command prompt you can do sudo -i. To run your above command you can simply do sudo vault. I suspect the actual issue is the use of ./pidfile where ./ is a relative path meaning "current directory". Systemd runs commands in it's own environment and the "current directory" may not be what you expect. Perhaps there is a way to tell vault where you want to store the pidfile with an absolute path?
    – Cliff Armstrong
    May 15 at 17:55
  • To get the same permissions you can do sudo -u vault vault where "-u vault" tells sudo to switch to the "vault" user to run the command instead of the root user. To get a command prompt as the user vault you can do sudo -i -u vault.
    – Cliff Armstrong
    May 15 at 18:02
  • Where is the configured value for pid_file in the Vault Agent configuration? If the configured location is within the vault user's home directory, then the issue is likely the systemd option ProtectHome=read-only.
    – GracefulRestart
    May 15 at 18:21
  • @CliffArmstrong About the pidfile, I changed for a absolute path in the config. Now I have Error storing PID: could not open pid file: open /etc/vault.d/pidfile: read-only file system which is, indeed, a but more clear, but still confusing since I have no issue running manually the command as a vault user or even editing files as vault in the exact same location where the pidfile is.
    – IlliciteS
    May 16 at 8:43
  • I've made an edit why I'm using the full sudo -s /bin/bash vault command. Also, I've tried to ExecStart=bin/sudo su -s /bin/bash vault -c "<command>" to have the same permissions but it did not like that : sudo: effective uid is not 0, is /bin/sudo on a file system with the 'nosuid' option set or an NFS file system without root privileges?
    – IlliciteS
    May 16 at 8:54

2 Answers 2

Reset to default
2

The option ProtectSystem=full literally mounts /etc as read-only for the process defined in the service:

Takes a boolean argument or the special values "full" or "strict". If true, mounts the /usr/ and the boot loader directories (/boot and /efi) read-only for processes invoked by this unit. If set to "full", the /etc/ directory is mounted read-only, too.

You should either move the pidfile to a writable location for that process, or remove the option ProtectSystem=full from the service file.

You should look into all of the other systemd service options that you are using which you are unsure of what they do. There are a number of other restrictions in there that may cause problems with your setup.

Improve this answer
1
  • That was that, you are right. I deleted the ProtectSystem=fullline from the service config and now it works, it can write the pid into /etc/vault.d/pidfile. Thank you very much!
    – IlliciteS
    May 17 at 7:54
1

It seems like vault is a script which launches a process, forks, saves the main PID to ./pidfile and exits.

First, this is an old-fashioned forking script. Therefore you need Type=forking in your service. When the initial vault process ends, systemd will know that it should follow another process instead of detecting a stopped or failed service.

But you don't get that far, so that's not your immediate issue.

The immediate problem is this program seems to write to ./pidfile. I doubt it's trying to write to /etc/vault.d/pidfile because:

  1. /etc is almost always read-only
  2. A PIDFile isn't really a configuration. It's a state, belonging in /run or /var.

Therefore we have to consider where it's really looking and really trying to create a file. Your error message says ./pidfile. By default, the working directory for systemd system services is /. Therefore ./pidfile resolves to /pidfile. You don't (and shouldn't) have write permission there.

Use WorkingDirectory=%t to set the working directory to /run (or maybe WorkingDirectory=%T for your private /tmp). Your service should have write permissions there.

Finally, Consider offering systemd a hint for tracking the main pid with PIDFile=%t/pidfile (or PIDFile=%T/pidfile).

I suspect /bin/vault is a script which sets some environment variables, runs your program, sets up logging, forks, and saves the pidfile like this:

IMPORTANT_VAR="important value"
/usr/lib/vault $@ | rotatelog /var/log/vault.log &
echo $! > ./pidfile

We really don't need forking processes anymore. All of that work can be delegated to systemd in a more standard way. A near equivalent would be to take advantage of the default Type=simple and call the process directly.

[Service]
Environment=IMPORTANT_VAR="important value"
ExecStart=/usr/lib/vault agent -non-interactive -config=/etc/vault.d/agent-config-prod.hcl
Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .