Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I have a group of programs that I was stress testing today. I drastically shot up the input load to the program and it worked at first, but I had an apparent memory leak sucking up memory. The only problem is that 1) Valgrind is insisting I don't have any leaks and 2) it only happens a high packet ingest rates.

One, possible crazy, idea was that the extra memory may be going towards a message queue due to my system lacking the bandwidth to output packets as fast as they are incoming forcing increasingly large buffers of data to be transmitted. The program that is showing as hogging the memory is reading from TCP socket and writing to a named pipe. so I have three questions.

  1. Can the in-memory queue for storing to-be dispatched data increase indefinately for either TCP/IP or named pipes?

  2. if the queue can and did grow would it still show that My program (the one reading from TCP and outputting to the named pipe) was the memory hog, or would the memory just show as system memory?

  3. is there a command I can use to check memory allocated to these queues? I'm running Centos

Admitedly I don't really think this is the cause of my leak, but then again I realy can't guess what else is so I have to ask!

share|improve this question
add comment

1 Answer

up vote 3 down vote accepted
  1. Generally its quite bounded. Though there are two places (at least) these buffers can exist:
    1. In the kernel. Definitely bounded. FIFO is hard-coded I believe, TCP can be set in /proc/sys/kernel/tcp_wmem (and rmem), and (within the proc limits) with setsockopt SO_SNDBUF/SO_RCVBUF. See the tcp(7) manpage for details.
    2. In your app. If you're not calling write, etc. directly, the library you're using may have its own buffers. These may be infinite.
    3. Ok. There is actually a third place, in RAM on your NIC. Can not be increased beyond a certain (fairly small) limit without soldering, so I'm ignoring this. Oh, and there is a transmit ring buffer, for your NIC to DMA out of. Again, fixed-size (ethtool to tune, if even possible).
  2. The kernel ones show as system memory. The ones in your app will show as part of your app's virtual size (and resident size, etc.)
  3. netstat -t will show you the current size of the send and receive queue for each TCP connection. For ones held in your library, check its documentation.

Remember, memory isn't leaked if you eventually free it. If you're just suffering from ever-growing queues in your app, that's not leaked. Same if you're suffering from ever-growing memory fragmentation.

share|improve this answer
    
I expected, and yet was hopping not, to get this answer. If it was the buffers I would at least have someplace to look at. Oh well. Thanks for your answer. –  dsollen Aug 29 '12 at 17:18
    
Does top show this as buffer-memory (sum)? –  Nils Aug 29 '12 at 21:35
    
@Nils I think buffer memory in top is filesystem buffers, not network. –  derobert Aug 29 '12 at 21:36
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.