Nginx leaves old socket

Question

I'm running Nginx 1.6.2 (the nginx-full package from the nginx/stable PPA). I'm using the unmodified configuration /etc/nginx/nginx.conf:

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 768;
}

http {
    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

If I make the following site configuration and link it to sites-enabled:

# /etc/nginx/sites-available/serve-files
server {
    listen unix:/run/serve-files.socket;
    root /var/www/files;
    location / {
        try_files $uri =404;
    }
}

And restart nginx (using sudo service nginx restart), the socket /run/serve-files.socket is created with the following permissions:

srw-rw-rw- 1 root root 0 Oct 29 14:35 serve-files.socket

If I then stop nginx (using sudo service nginx stop), the socket unexpectedly remains. And when I start nginx back up (using sudo service nginx start), I get the following errors reported to /var/log/nginx/error.log:

2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: still could not bind()

It appears that nginx will not overwrite its socket that was left from its previous shutdown. Why is this? Have I misconfigured something? Is there a way to work around this?

NOTE: There are no other sites running with nginx, when I stop nginx there are no lingering processes, and I've reproduced this on an Ubuntu server and desktop both running 14.04.1 LTS.

UPDATE: When nginx is running, netstat -lx | grep serve-files will indicate the socket is being used:

unix  2      [ ACC ]     STREAM     LISTENING     6543310  /run/serve-files.socket

When nginx is stopped, netstat -lx | grep serve-files indicates no socket is being used (as expected) but the socket file remains at /run/serve-files.socket.

Answer 1

According to the Nginx documentation, SIGQUIT will perform a "graceful shutdown" while SIGTERM will perform a "fast shutdown". At least as of version 1.8.0, Nginx will leave stale UNIX domain sockets when it is stopped using the SIGQUIT signal. However, the UNIX domain sockets are properly removed when using the SIGTERM signal.

The Nginx service script /etc/init.d/nginx provided from the nginx/stable PPA sends SIGQUIT to Nginx when it is stopped with sudo service nginx stop or restart. To patch the script, modify the line:

STOP_SCHEDULE="${STOP_SCHEDULE:-QUIT/5/TERM/5/KILL/5}"

To:

STOP_SCHEDULE="${STOP_SCHEDULE:-TERM/5/KILL/5}"

However, the Nginx service script from the Ubuntu repo already uses SIGTERM instead of SIGQUIT and does not need to be modified.

Answer 2

I don't think it has to do with graceful or fast shutdown, since I've been using graceful stop and I'm also having this problem.

In my case, I've noticed the possibly related/additional problem that nginx continues to try to connect to a long-since-defunct socket, even after the socket and all the calls to it have been deleted (at least so far as files that I deliberately created or altered).

Putting the socket in /tmp seems to be a workaround that will avoid this problem, but I still can't get rid of the ghost sockets that were previously created elsewhere.

So I don't understand where the problem is even coming from. At what point did nginx decide to remember a socket forever, and is there a way to reset that?

Answer 3

As a quick fix on 16.04 Xenial I was simply able to delete the offending socket:

sudo systemctl stop nginx
sudo rm /var/run/serve-files.socket
sudo systemctl start nginx