How to start a systemd user unit over ssh connection with correct environment variables?

Question

I have a systemd --user unit that runs correctly when run from the machine's console (Arch Linux). However, when I ssh into the machine and run the same command to start the user unit, it fails. The errors are:

[user@remote ~]$ systemctl --user start my-user-service.service  
Failed to connect to bus: No such file or directory

[user@remote ~]$ systemctl --user show-environment  
Failed to connect to bus: No such file or directory

I have UsePAM yes set in sshd_config.

If I add the following 2 lines to the user's .bashrc on the remote machine (before connecting an SSH session as that user), I can work around the problem.

export XDG_RUNTIME_DIR="/run/user/$UID"
export DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus"

However, that method is not mentioned in the Arch Wiki, therefore I assume using it is not recommended. I tried all the methods mentioned at systemd/User#Environment_variables including writing an environment generator script. I could not get any of them to work. In each case, it is the same error first shown above.

Here's a simple generator I tried: /usr/lib/systemd/user-environment-generators/40-user-env-generator

#!/bin/bash

echo XDG_RUNTIME_DIR="/run/user/$UID"
echo DBUS_SESSION_BUS_ADDRESS="unix:path=${XDG_RUNTIME_DIR}/bus"

What would be the correct way to resolve this?

EDIT: added new info:

cat /etc/pam.d/sshd

#%PAM-1.0
#auth     required  pam_securetty.so     #disable remote root
auth      include   system-remote-login
account   include   system-remote-login
password  include   system-remote-login
session   include   system-remote-login

cat /etc/pam.d/system-remote-login

#%PAM-1.0
auth      include   system-login
account   include   system-login
password  include   system-login
session   include   system-login

cat /etc/pam.d/system-login

#%PAM-1.0
auth       required   pam_tally2.so        onerr=succeed file=/var
/log/tallylog
auth       required   pam_shells.so
auth       requisite  pam_nologin.so
auth       include    system-auth

account    required   pam_tally2.so
account    required   pam_access.so
account    required   pam_nologin.so
account    include    system-auth

password   include    system-auth

session    optional   pam_loginuid.so
session    optional   pam_keyinit.so       force revoke
session    include    system-auth
session    optional   pam_motd.so          motd=/etc/motd
session    optional   pam_mail.so          dir=/var/spool/mail standard quiet
-session   optional   pam_systemd.so
session    required   pam_env.so

cat /etc/pam.d/system-auth

#%PAM-1.0

auth      required  pam_unix.so     try_first_pass nullok
auth      optional  pam_permit.so
auth      required  pam_env.so

account   required  pam_unix.so
account   optional  pam_permit.so
account   required  pam_time.so

password  required  pam_unix.so     try_first_pass nullok sha512 shadow
password  optional  pam_permit.so

session   required  pam_limits.so
session   required  pam_unix.so
session   optional  pam_permit.so

Answer 1

Check the PAM settings for sshd, i.e. /etc/pam.d/sshd and any file it refers to using the @include keyword.

There should be a pam_systemd.so module invoked in the session phase, like this:

session optional pam_systemd.so

Among other things, this PAM module will initialize the two environment variables XDG_RUNTIME_DIR and DBUS_SESSION_BUS_ADDRESS.

Answer 2

Notice: If everything is fine locally then the problem is connected with the other thing than *-environment-generators.

For a workaround from the question I suggest you replace $UID with $(id -ru) (because the env variable can be not set and use not user-environment-generators but system-environment-generators ( see https://man.archlinux.org/man/systemd.environment-generator.7 ) because the daemon itself for systemctl --user is run as a system service so it is affected by the system environment.

Answer 3

Make sure that the SSH server is configured to use PAM modules. In /etc/ssh/sshd_config or /etc/ssh/sshd_config.d/99-local.conf set

UsePAM yes