I have an nginx service that resides on ports 80 and 443 and has done som for years. Yesterday I suddenly just got a plain text page with a 404 page not found message
Headers:
HTTP/2 404
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Sun, 01 Jan 2023 10:24:28 GMT
In order to troubleshoot, I started a terminal on another pc and started watch curl -ksI https://trausti.local
The output of that terminal window remained unchanged, even though i stopped the nginx service sudo service nginx stop. I verifíed this with
to figure out which other process might have overtaken the port, I ran these commands:
netstat -tulpn | grep :80
$ netstat -tulpn | grep :80 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8086 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN -
tcp6 0 0 :::8086 :::* LISTEN -
tcp6 0 0 :::8088 :::* LISTEN -
ss -lptr
Both with no indication of the culprit...
I verified that the nginx service was stopped with sudo systemctl status nginx:
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2023-01-01 11:15:05 CET; 5min ago
Docs: man:nginx(8)
Process: 572 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 935 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 124643 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exite>
Main PID: 936 (code=exited, status=0/SUCCESS)
CPU: 374ms
Jan 01 00:45:13 Trausti systemd1: Starting A high performance web server and a reverse proxy server...
Jan 01 00:45:22 Trausti systemd1: Started A high performance web server and a reverse proxy server.
Jan 01 11:15:03 Trausti systemd1: Stopping A high performance web server and a reverse proxy server...
Jan 01 11:15:05 Trausti systemd1: nginx.service: Succeeded.
Jan 01 11:15:05 Trausti systemd1: Stopped A high performance web server and a reverse proxy server.
The only new service I have installed recently is k3s and that is a couple of days ago.
I restarted the server completely at 00:45 which returned the normal nginx webpage, but now it is back at 404 page not found
uname -a output:
Linux Trausti 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux
Edit:
as @steve suggested it was related to k3s. Specifically the Traefik Ingress controller and reverse proxy I am still very curious as to how it manages to hijack the ports withyout anything showing up. (and iritated that there were NO HEADERS WHATSOEVER indicating which service it is).
I still wonder at how it can listen on port 80 and 443 without turning up in ss or netstat
sudo netstat -alntp|grep ':80'say? any help about the process there?k3shad by e.g. using iptables DNAT or REDIRECT targets to reroute connections incoming to a particular port towards a new target. Since all this happens below the application protocol layer (HTTP/2), there is no inherent requirement to generate any headers or other indications, as these manipulations are intended to be not detectable by the applications whose traffic is being manipulated.