I run on a RPi (Raspbian
3.12.35+ #730 PREEMPT Fri Dec 19 18:31:24 GMT 2014 armv6l GNU/Linux) a daemon in Python 2.7.8 which queries various APIs via
requests and provides itself an API for another application which makes use of it (both run on the same machine). I found out recently in the logs (after the daemon has been running for some time) that I ran out of open file handlers via a
('Connection aborted.', error(23, 'Too many open files in system'))
Since I do not explicitly open any files I suspected my
http calls to be left open.
I make a call every about 10 seconds (calls to localhost for the most part) and after running the daemon for an hour or two
lsof reports 32k+ open
python2 804 pi 982u unix 0xce3565a0 0t0 116809 socket python2 804 pi 983u unix 0xcb5553c0 0t0 116875 socket python2 804 pi 984u unix 0xcb555780 0t0 116899 socket python2 804 pi 985u unix 0xd74dd1e0 0t0 117010 socket python2 804 pi 986u unix 0xcb555d20 0t0 116918 socket
This number increases more or less in line with the rate of the calls, all applications together.
I wanted to force the close these connections by explicitly setting
'Connection': 'close' in my code but unfortunately this does not help,
lsof still shows the myriad of open
netstat shows a huge amount of
DGRAM connections (about 12k+):
(...) unix 2 [ ] DGRAM 74315 unix 2 [ ] DGRAM 68340 unix 2 [ ] DGRAM 67749 unix 2 [ ] DGRAM 58836 unix 2 [ ] DGRAM 58087 (...)
What worries me even more is that it looks like the tcp sequence seems to close correctly, at a network level:
So I am quite lost here. The growing number of open descriptors is likely due to the http calls but I have a hard time understanding why they leave behind open handlers?
Note: I initially asked this question on SO (pointing at my code) but I feel it is more system-oriented.