Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm running Sphinx Search Server V2.06 ( the latest stable release ) on an Amazon EC2 with Linux CentOS. In general it is running fine, but the searchd.log is showing this error over and over with varying number of repeat times:

send() failed: 32: Broken pipe
WARNING: last message repeated 6 times

I assume this is some sort of lost connection ( based on some Sphinx forum responses ). I'd like to fix this, but it's not our primary concern ... but possibly related. After Sphinx has been running for a while, or under heavy load ( as near as I can tell ), the number of times the 'message repeats' goes up and peaks at 100. Generally around the same time the following fatal error will occur and it will shut Sphinx down:

FATAL: accept() failed: Too many open files

I've looked into upping the file limits on my system but am not sure what exactly to do. Here is what my system currently reports. I am still seeing this error.

sysctl fs.file-max ... returns ... fs.file-max = 7017952
ulimit -a ... returns ... open files 1024
ulimit -Hn ... returns ... 4096
ulimit -Sn ... returns ... 1024

I don't really know what these different numbers mean, but I think they could be used to solve my problem according to this article. How can I fix the sphinx Fatal error and make sure that the system maintains this 'fixed' configuration after a reboot?

share|improve this question

1 Answer 1

Let's start with some helpful articles

Plus the one you have already listed.

Basically what it tells you is this:

  • Maxmimum number of open file descriptors per system: 7017952
  • ulimit -a: Maximum number of file descriptors allowed to be opened by a shell and processes it starts
  • ulimit -Sn: Same as above but show only soft limit for maximum number of file descriptors
  • ulimit -Hn: Shows hard limit for session open file descriptors

Basically what you need to do is look at output of lsof for your process to see where it's getting stuck. Soft limit you can change up or down to change the possible number of open file descriptors during the session. Hard limit you can only lower but and only root can increase.

So I would suggest looking at:

sysctl fs.file-nr

which would give you the total number of file descriptors open and unused on the system and also output of

lsof -p <pid> 

where <pid> is the process in question to determine how many files and sockets that process has open and see if you are hitting your limit.

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.