3

We have a webserver that performs scientific calculations submitted by users. The calculations can be long-running, so we use The Torque resource manager (aka pbs_server) to distribute/schedule them on a handful of compute nodes. Torque makes use of a unix domain socket in the /tmp directory for communication but the http server (and processes forked from it) can't access the true /tmp directory, so to those processes, the socket appears to be missing, resulting in an error.

The Details:

  • The webserver is running Apache, which runs as a service with the systemd property PrivateTmp=true set. This casuses the service to have its own /tmp directory unrelated to the "true" root /tmp.
  • The jobs are actually submitted from PHP (running in the Apache process). PHP makes a system call to qsub, which is a Torque command to submit a job. Because qsub is called from PHP, it inherits the "fake" /tmp directory from Apache.
  • qsub internally attempts to connect to the unix socket located at /tmp/trqauthd-unix. But since it doesn't see the real /tmp directory, it fails with the following error: Error in connection to trqauthd (15137)-[could not connect to unix socket /tmp/trqauthd-unix: 2]

The only solution I could achieve was to edit the httpd.service file under systemd and change PrivateTmp to false. This DID fix the problem. But I'd rather not do this because (I assume) PrivateTmp was set to true for good reason.

What I want to know is whether there is any way to have the socket created in a different location or to somehow make a link to the socket that could be used from within Apache (and its forked processes).

Creating a link to the socket is trivial, but it doesn't solve the problem because I don't know of any way to configure qsub to look for the socket in a different location.

Note that the socket is created by the trqauthd service (a Torque program that performs user authorization for running jobs). The documentation for trqauthd mentions (in an obscure note) that the location of the socket can be configured, but there is no indication in any of the documentation about how that can be achieved (and more importantly, how to let qsub and other commands know about the new location).

Thanks for any suggestions that might help me find a way to submit jobs to Torque from PHP without disabling PrivateTmp for Apache.

4

In 2013, trqauthd stopped using IP sockets and switched to a Unix domain socket in the server's home directory.

Later that same year, trqauthd switched from the home directory to /tmp.

As you can see, the only option that Adaptive Computing has given to you for altering /tmp/trqauthd-unix is to re-compile the programs from source, changing the --with-trqauthd-sock-dir build configuration option to denote somewhere other than /tmp. (/run/trqauthd perhaps?)

  • That's what I figured. So basically, the only options for getting this to work are (1) set PrivateTmp=false in httpd.service or (2) Recompile Torque from source. Option 1 potentially adds security vulnerabilities and Option 2 makes maintenance a problem, because pre-built binary packages are no longer an option, and each subsequent sys-admin has to make special note of this or spend hours trying to find out why /tmp/trqauth-unix is unaccessible to PHP despite all the standard unix permissions being right. – drwatsoncode Nov 10 '16 at 18:19
  • I'll add this just as a comment, but a third option might be to write small daemon that would simply watch a folder for job scripts and submit them to qsub when present. That way, PHP could just worry about writing the script and qsub would be invoked by the daemon (which would NOT be running under the Apache process and hence would not be using a private /tmp folder. But I'm not sure that this doesn't add more security holes than simply turning PrivateTmp=false for Apache. – drwatsoncode Nov 10 '16 at 18:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.