How can I find a memory leak of a running process?

Question

Is there a way, I can find the memory leak of a running process? I can use Valgrind for finding memory leaks before the start of a process. I can use GDB to attach it to a running process. How could I debug a memory leaks of a running process?

Answer 1

Here are the steps that almost guarantee to find what is leaking memory:

1. Find out the PID of the process which causing memory leak.

   ps -aux
   

2. capture the /proc/PID/smaps and save into some file like BeforeMemInc.txt.

3. wait till memory gets increased.
4. capture again /proc/PID/smaps and save it has afterMemInc.txt

5. find the difference between first smaps and 2nd smaps, e. g. with

   diff -u beforeMemInc.txt afterMemInc.txt

6. note down the address range where memory got increased, for example:

      beforeMemInc.txt            afterMemInc.txt
   ---------------------------------------------------
   2b3289290000-2b3289343000   2b3289290000-2b3289343000  #ADDRESS
   Shared_Clean:    0 kB       Shared_Clean:    0 kB          
   Shared_Dirty:    0 kB       Shared_Dirty:    0 kB
   Private_Clean:   0 kB       Private_Clean:   0 kB
   Private_Dirty:  28 kB       Private_Dirty:  36 kB  
   Referenced:     28 kB       Referenced:     36 kB
   Anonymous:      28 kB       Anonymous:      36 kB  #INCREASE MEM
   AnonHugePages:   0 kB       AnonHugePages:   0 kB
   Swap:            0 kB       Swap:            0 kB
   KernelPageSize:  4 kB       KernelPageSize:  4 kB
   MMUPageSize:     4 kB       MMUPageSize:     4 kB
   Locked:          0 kB       Locked:          0 kB
   VmFlags: rd wr mr mw me ac  VmFlags: rd wr mr mw me ac
   

7. use GDB to dump memory on running process or get the coredump using gcore -o process

8. I used gdb on running process to dump the memory to some file.

   gdb -p PID
   dump memory ./dump_outputfile.dump 0x2b3289290000 0x2b3289343000
   

9. now, use strings command or hexdump -C to print the dump_outputfile.dump

   strings outputfile.dump
   

10. You get readable form where you can locate those strings into your source code.

11. Analyze your source to find the leak.

Answer 2

I think memleax is exact what you want.

It debugs memory leak of a running process by attaching it, without recompiling program or restarting target process. It's very convenient and suitable for production environment.

It works on GNU/Linux and FreeBSD.

NOTE: I'm the author, any suggestion is welcome.

== EDIT ==

I also wrote another tool libleak, which hooks memory functions by LD_PRELOAD.

There is no need to modify the target program. Although you have to restart the progress with LD_PRELOAD, you can enable/disable the detection during running.

There is much less impact on performance since no signal trap.

Compared with similar tools (such as mtrace), it print the full call-stack at suspicious memory leak point.

Answer 3

On Linux, you could enable mtrace in your program, but it is a code change.

On OpenBSD, you could try the malloc stats.

Google's leak checker might also be worth a look, and unlike mtrace you may be able to use LD_PRELOAD to avoid recompilation.

Answer 4

I think without providing support for allocation monitoring after program start directly in the source code, you're out of luck. Here are two reasons I can think of:

• Heap checkers initialize when the program begins. Some offer the ability to tweak the exact timing, but the environment variables that start them must be set when the program runs. This is because they watch to make sure each allocation has a corresponding deallocation, and they would miss some otherwise.
• Heap checking usually requires elevated privileges, or hooks, to be provided by the operating system. If those hooks aren't provided at the time of program start, the heap checkers cannot leverage them. I don't believe OSes provide these privileges after the program in question is started.

If, however, you're program is running inside a virtual machine, that environment may provide support for monitoring allocations. I know Java has several allocation and garbage collection monitoring tools (like visualVM) that attach to running programs or VMs.

Answer 5

IBM's Purify is probably the oldest and most sophisticated tool of all. It will flag the line number in the code which causes the memory leak.