1

This is my first post in this forum, and I was compelled to come here due to the nature of this issue. Both my boss and I are stumped on what could be going on, and thus had to come to the experts here.

As a preface, I am the junior web developer at a small company, and do not have direct access to the company's server configurations. All of my work is done locally on docker and pushed to my boss for approval. If you need any exact details of our setup, I'll happy update the post with anything you all need.

Anyway, as the title of this post indicates, we have been slowly bringing our web apps from Debian Stretch with Apache2 to Debian Bullseye with Apache 2.

We have 3 web apps running in this server environment, 2 are PHP and 1 is Perl. All 3 have file uploading capabilities for our users.

To add to the possible complexity, my boss was running three web servers. web01 and web02 are running the working Stretch version. He was then trying to launch our Bullseye update to web03 for a soft deployment and minor user testing.

Now, me being the jr developer that I am, completely glossed over this bug when testing in my local docker environment. My boss ended up finding this upload issue after deployment to web03. Note, that this upload issue is afflicting both our PHP and Perl web apps, so it seems to be configuration related, rather than code related. Everything else in the Bullseye deployment is working beautifully, but the file upload and retrieval is the only bug afflicting us, and I can't find anyone else with this specific issue.

Below is just a simple code snippet from our Perl program. It loves to fail at this line, and claim it can't fine the /tmp directory, despite existing.

open(my $fh, "<", "/tmp/$csvfile") or die "Can't open $csvfile: $!";

Here's a direct quote from my boss describing what he sees going on his end.

Got a weird issue that it can't find the files uploaded to /tmp. I know systemd does weird stuff with apache2 /tmp and points it to a separate private folder, but that was already the case in stretch. Something in bullseye seems to be breaking it more. It's not perl or php specific since both the app 1 and app 2 are affected. jobtracker gives a "cannot find file /tmp/xxxxx.csv" error. app 2 complains about insufficient disk space, but that's just a default error because it can't determine the filesize being uploaded.

I can clearly see the timestamp on the /tmp/systemd-..../tmp/ path is being updated when I upload something, so it's doing something, I just don't know where it's failing.

I am genuinely at my limit of back end knowledge here, but I'd really like to help my boss, and get some brownie points by saving the day.

Please let me know if there would be a more appropriate place to post this, but my gut is telling me its a Debian or Apache2 configuration issue if two apps in two different languages are both afflicted with the same file upload issue.

My boss initially thought it could have been a load balancing issue, since all three web servers are running at the same time, but he recreated the upload issue even with web03 running independently from our web01 and 02 environments. I was also getting the same bug on my docker environment as well, which should be operating in its own localized environment aside from database access, right?

If anyone needs any more context, please don't hesitate to ask. I'll try my best to provide context.

0

1 Answer 1

1

Debian 11's apache settings when run by systemd are in /lib/systemd/system/apache2.service and include in the [Service] section: PrivateTmp=true, which is documented as:

PrivateTmp=

Takes a boolean argument. If true, sets up a new file system namespace for the executed processes and mounts private /tmp/ and /var/tmp/ directories inside it that are not shared by processes outside of the namespace. This is useful to secure access to temporary files of the process, but makes sharing between processes via /tmp/ or /var/tmp/ impossible. If true, all temporary files created by a service in these directories will be removed after the service is stopped. Defaults to false.
[...]

So apache2's /tmp/ is hidden and isn't the same as initial host's /tmp/.

Of course one could override the setting (using sysctl edit apache2 and adding a Service section with PrivateTmp=false), but this wouldn't be advised: a private /tmp is there for additional security, and it's quite possible that other packaged applications would also have the same setting and would also have to be changed (doesn't appear to be the case for php-fpm). Better have a dedicated directory for sharing this and reconfigure applications accordingly.

If it's just to debug or simply figure out where are these hidden files, then there's a shortcut to access a process' mount namespace without having to use unshare -m which is often difficult for these cases. /proc/<pid>/root/ is a direct link to this process mount's namespace / mountpoint. For example if pgrep -u root ^apache2$ gives one PID result (or if one picks any spawned apache2 PID at random), eg 12345, then ls /proc/12345/root/tmp/ will display the content of this now hidden /tmp/. It's also possible to copy files to or from there this way, but this shouldn't be used for anything serious. This is documented in man proc.

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .