I run on a RPi (Raspbian 3.12.35+ #730 PREEMPT Fri Dec 19 18:31:24 GMT 2014 armv6l GNU/Linux) a daemon in Python 2.7.8 which queries various APIs via requests and provides itself an API for another application which makes use of it (both run on the same machine). I found out recently in the logs (after the daemon has been running for some time) that I ran out of open file handlers via a requests exception:

('Connection aborted.', error(23, 'Too many open files in system'))

Since I do not explicitly open any files I suspected my http calls to be left open.

I make a call every about 10 seconds (calls to localhost for the most part) and after running the daemon for an hour or two lsof reports 32k+ open sock connections:

python2     804             pi  982u     unix 0xce3565a0        0t0     116809 socket
python2     804             pi  983u     unix 0xcb5553c0        0t0     116875 socket
python2     804             pi  984u     unix 0xcb555780        0t0     116899 socket
python2     804             pi  985u     unix 0xd74dd1e0        0t0     117010 socket
python2     804             pi  986u     unix 0xcb555d20        0t0     116918 socket

This number increases more or less in line with the rate of the calls, all applications together.

I wanted to force the close these connections by explicitly setting 'Connection': 'close' in my code but unfortunately this does not help, lsof still shows the myriad of open sock handlers.

netstat shows a huge amount of DGRAM connections (about 12k+):

unix  2      [ ]         DGRAM                    74315
unix  2      [ ]         DGRAM                    68340
unix  2      [ ]         DGRAM                    67749
unix  2      [ ]         DGRAM                    58836
unix  2      [ ]         DGRAM                    58087

What worries me even more is that it looks like the tcp sequence seems to close correctly, at a network level:

enter image description here

So I am quite lost here. The growing number of open descriptors is likely due to the http calls but I have a hard time understanding why they leave behind open handlers?

Note: I initially asked this question on SO (pointing at my code) but I feel it is more system-oriented.

  • 1
    Do you have a link to the SO question/the code in question? – Pod Oct 28 '16 at 12:55
  • 3
    The first thing that comes to mind is that this is http 1.1 and therefore you have en.wikipedia.org/wiki/HTTP_persistent_connection – icarus Oct 29 '16 at 5:45
  • Sorry for stating the obvious, but maybe you can close these explicitly in your program (after use), like s.close()? Unless you are already doing that but it is ignored somehow? – Ned64 Nov 4 '16 at 7:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.