3

I'm running Nginx 1.6.2 (the nginx-full package from the nginx/stable PPA). I'm using the unmodified configuration /etc/nginx/nginx.conf:

user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
    worker_connections 768;
}

http {
    ##
    # Basic Settings
    ##

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;

    include /etc/nginx/mime.types;
    default_type application/octet-stream;

    ##
    # Logging Settings
    ##

    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;

    ##
    # Gzip Settings
    ##

    gzip on;
    gzip_disable "msie6";

    ##
    # Virtual Host Configs
    ##

    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

If I make the following site configuration and link it to sites-enabled:

# /etc/nginx/sites-available/serve-files
server {
    listen unix:/run/serve-files.socket;
    root /var/www/files;
    location / {
        try_files $uri =404;
    }
}

And restart nginx (using sudo service nginx restart), the socket /run/serve-files.socket is created with the following permissions:

srw-rw-rw- 1 root root 0 Oct 29 14:35 serve-files.socket

If I then stop nginx (using sudo service nginx stop), the socket unexpectedly remains. And when I start nginx back up (using sudo service nginx start), I get the following errors reported to /var/log/nginx/error.log:

2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: bind() to unix:/run/serve-files.socket failed (98: Address already in use)
2014/10/29 14:36:32 [emerg] 21680#0: still could not bind()

It appears that nginx will not overwrite its socket that was left from its previous shutdown. Why is this? Have I misconfigured something? Is there a way to work around this?

NOTE: There are no other sites running with nginx, when I stop nginx there are no lingering processes, and I've reproduced this on an Ubuntu server and desktop both running 14.04.1 LTS.

UPDATE: When nginx is running, netstat -lx | grep serve-files will indicate the socket is being used:

unix  2      [ ACC ]     STREAM     LISTENING     6543310  /run/serve-files.socket

When nginx is stopped, netstat -lx | grep serve-files indicates no socket is being used (as expected) but the socket file remains at /run/serve-files.socket.

  • Unles I'm misunderstanding, nginx is listening on that socket, it doesn't create it. Why would you expect it to remove it? – Chris Down Oct 29 '14 at 20:21
  • @ChrisDown Nginx opens/creates the socket so that it can listen. – user369450 Oct 29 '14 at 20:47
  • I'm not sure if that's a typical way of operating, typically the socket is external. It wouldn't surprise me if nginx doesn't consider itself responsible for removing the socket after terminating. – Chris Down Oct 29 '14 at 21:26
  • @ChrisDown I'm confused. If nginx creates the socket, why would it not be responsible for removing it? It's not connecting to a socket managed by a separate process. – user369450 Oct 29 '14 at 22:01
  • My point is that I don't think having nginx create the socket is a typical outcome of the listen directive, so it probably doesn't have code to remove the socket on exit. Typically, you have some other application that creates the socket, and nginx just listens on it. It probably doesn't remove it since it doesn't know what the other application wants to do with it. – Chris Down Oct 29 '14 at 22:04
3

According to the Nginx documentation, SIGQUIT will perform a "graceful shutdown" while SIGTERM will perform a "fast shutdown". At least as of version 1.8.0, Nginx will leave stale UNIX domain sockets when it is stopped using the SIGQUIT signal. However, the UNIX domain sockets are properly removed when using the SIGTERM signal.

The Nginx service script /etc/init.d/nginx provided from the nginx/stable PPA sends SIGQUIT to Nginx when it is stopped with sudo service nginx stop or restart. To patch the script, modify the line:

STOP_SCHEDULE="${STOP_SCHEDULE:-QUIT/5/TERM/5/KILL/5}"

To:

STOP_SCHEDULE="${STOP_SCHEDULE:-TERM/5/KILL/5}"

However, the Nginx service script from the Ubuntu repo already uses SIGTERM instead of SIGQUIT and does not need to be modified.

  • Thank you! I'd note that to fix nginx as currently provided by "nginx stable PPA"—1.12.2-0—one need to patch the "stop" action in its upstart file by changing the parameter passed to /usr/bin/nginx -s from quit to stop (the latter sends SIGINT—see man nginx). – kostix Feb 2 '18 at 19:13
  • 2
    The relevant ticket in the nginx bug tracker is #753. – kostix Feb 2 '18 at 19:14
-1

I don't think it has to do with graceful or fast shutdown, since I've been using graceful stop and I'm also having this problem.

In my case, I've noticed the possibly related/additional problem that nginx continues to try to connect to a long-since-defunct socket, even after the socket and all the calls to it have been deleted (at least so far as files that I deliberately created or altered).

Putting the socket in /tmp seems to be a workaround that will avoid this problem, but I still can't get rid of the ghost sockets that were previously created elsewhere.

So I don't understand where the problem is even coming from. At what point did nginx decide to remember a socket forever, and is there a way to reset that?

-1

As a quick fix on 16.04 Xenial I was simply able to delete the offending socket:

sudo systemctl stop nginx
sudo rm /var/run/serve-files.socket
sudo systemctl start nginx
  • Well... yes, you can manually delete the socket. The question was, "Why does the socket remain?". – user369450 Jun 27 '18 at 18:34
  • @cpburnz where does it say "why does the socket remain?" because I am providing an answer to the question "Is there a way to work around this?" which is explicitly asked in the question text. Downvote the question if you think it is too broad/asking too many questions -- not a valid answer.I also wonder if you think most people happening upon this page from googling an error will be concerned in a practical solution or a forensic analysis of NGiNX internals... – Pocketsand Jun 28 '18 at 11:39
  • I didn't literally say "why does the socket remain" in the question, but that is essentially what I was asking (I shouldn't have quoted it in my previous comment). In my case I was concerned about how to fix the underlying issue (why is nginx not removing the socket), not the just symptom (delete the old socket file every time you restart nginx) – user369450 Jun 28 '18 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.