We have a webserver that performs scientific calculations submitted by users. The calculations can be long-running, so we use The Torque resource manager (aka pbs_server) to distribute/schedule them on a handful of compute nodes. Torque makes use of a unix domain socket in the
/tmp directory for communication but the http server (and processes forked from it) can't access the true
/tmp directory, so to those processes, the socket appears to be missing, resulting in an error.
- The webserver is running Apache, which runs as a service with the systemd property
PrivateTmp=trueset. This casuses the service to have its own
/tmpdirectory unrelated to the "true" root /tmp.
- The jobs are actually submitted from PHP (running in the Apache process). PHP makes a system call to
qsub, which is a Torque command to submit a job. Because
qsubis called from PHP, it inherits the "fake"
/tmpdirectory from Apache.
qsubinternally attempts to connect to the unix socket located at
/tmp/trqauthd-unix. But since it doesn't see the real /tmp directory, it fails with the following error:
Error in connection to trqauthd (15137)-[could not connect to unix socket /tmp/trqauthd-unix: 2]
The only solution I could achieve was to edit the httpd.service file under systemd and change
PrivateTmp to false. This DID fix the problem. But I'd rather not do this because (I assume) PrivateTmp was set to true for good reason.
What I want to know is whether there is any way to have the socket created in a different location or to somehow make a link to the socket that could be used from within Apache (and its forked processes).
Creating a link to the socket is trivial, but it doesn't solve the problem because I don't know of any way to configure
qsub to look for the socket in a different location.
Note that the socket is created by the
trqauthd service (a Torque program that performs user authorization for running jobs). The documentation for trqauthd mentions (in an obscure note) that the location of the socket can be configured, but there is no indication in any of the documentation about how that can be achieved (and more importantly, how to let qsub and other commands know about the new location).
Thanks for any suggestions that might help me find a way to submit jobs to Torque from PHP without disabling PrivateTmp for Apache.