2

I have an nginx service that resides on ports 80 and 443 and has done som for years. Yesterday I suddenly just got a plain text page with a 404 page not found message

Headers:

HTTP/2 404
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
content-length: 19
date: Sun, 01 Jan 2023 10:24:28 GMT

In order to troubleshoot, I started a terminal on another pc and started watch curl -ksI https://trausti.local

The output of that terminal window remained unchanged, even though i stopped the nginx service sudo service nginx stop. I verifíed this with

to figure out which other process might have overtaken the port, I ran these commands:

netstat -tulpn | grep :80

$ netstat -tulpn | grep :80 (Not all processes could be identified, non-owned process info will not be shown, you would have to be root to see it all.)
tcp 0 0 0.0.0.0:8086 0.0.0.0:* LISTEN -
tcp 0 0 0.0.0.0:8088 0.0.0.0:* LISTEN -
tcp6 0 0 :::8086 :::* LISTEN -
tcp6 0 0 :::8088 :::* LISTEN -

ss -lptr

Both with no indication of the culprit...

I verified that the nginx service was stopped with sudo systemctl status nginx:

● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sun 2023-01-01 11:15:05 CET; 5min ago
Docs: man:nginx(8)
Process: 572 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 935 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 124643 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exite>
Main PID: 936 (code=exited, status=0/SUCCESS)
CPU: 374ms

Jan 01 00:45:13 Trausti systemd1: Starting A high performance web server and a reverse proxy server...
Jan 01 00:45:22 Trausti systemd1: Started A high performance web server and a reverse proxy server.
Jan 01 11:15:03 Trausti systemd1: Stopping A high performance web server and a reverse proxy server...
Jan 01 11:15:05 Trausti systemd1: nginx.service: Succeeded.
Jan 01 11:15:05 Trausti systemd1: Stopped A high performance web server and a reverse proxy server.

The only new service I have installed recently is k3s and that is a couple of days ago.

I restarted the server completely at 00:45 which returned the normal nginx webpage, but now it is back at 404 page not found

uname -a output:
Linux Trausti 5.10.0-20-amd64 #1 SMP Debian 5.10.158-2 (2022-12-13) x86_64 GNU/Linux

Edit:

as @steve suggested it was related to k3s. Specifically the Traefik Ingress controller and reverse proxy I am still very curious as to how it manages to hijack the ports withyout anything showing up. (and iritated that there were NO HEADERS WHATSOEVER indicating which service it is).

I still wonder at how it can listen on port 80 and 443 without turning up in ss or netstat

7
  • what does sudo netstat -alntp|grep ':80' say? any help about the process there?
    – stoney
    Jan 1, 2023 at 10:48
  • @stoney just another couple of processes listening on 8088 and 8086
    – JoSSte
    Jan 1, 2023 at 10:52
  • You should run this as root, you might not see all procesws info otherwise (as netstat stated). On a side note: could it be some kind of proxy that is interfering?
    – stoney
    Jan 1, 2023 at 11:10
  • sudo does run it as root ;-)
    – JoSSte
    Jan 1, 2023 at 12:40
  • 1
    You could have an effect that's basically similar to what k3s had by e.g. using iptables DNAT or REDIRECT targets to reroute connections incoming to a particular port towards a new target. Since all this happens below the application protocol layer (HTTP/2), there is no inherent requirement to generate any headers or other indications, as these manipulations are intended to be not detectable by the applications whose traffic is being manipulated.
    – telcoM
    Jan 1, 2023 at 15:01

1 Answer 1

3

It's because k3s ingress is forwarding incoming port 80 to some other port.

k3s has setup a firewall rule so that anything targetting port 80 gets sent to another port, which k3s is then listening on.

As designed.

3
  • How can it take over the port when nginx had it, and why does starting nginx not cause a port in use error?
    – JoSSte
    Jan 1, 2023 at 12:40
  • besides that sudo service stop k3s-agent does not make the 404 page go away
    – JoSSte
    Jan 1, 2023 at 12:55
  • but sudo k3s-killall.sh did make the 404 page disappear
    – JoSSte
    Jan 1, 2023 at 13:03

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .