Is there a way, I can find the memory leak of a running process? I can use Valgrind for finding memory leaks before the start of a process. I can use GDB to attach it to a running process. How could I debug a memory leaks of a running process?
-
Valgrind is very useful, I would even call it intuitive.– user400344Jul 15, 2016 at 9:23
5 Answers
Here are the steps that almost guarantee to find what is leaking memory:
Find out the PID of the process which causing memory leak.
ps -aux
capture the
/proc/PID/smaps
and save into some file likeBeforeMemInc.txt
.- wait till memory gets increased.
- capture again
/proc/PID/smaps
and save it hasafterMemInc.txt
find the difference between first
smaps
and 2ndsmaps
, e. g. withdiff -u beforeMemInc.txt afterMemInc.txt
note down the address range where memory got increased, for example:
beforeMemInc.txt afterMemInc.txt --------------------------------------------------- 2b3289290000-2b3289343000 2b3289290000-2b3289343000 #ADDRESS Shared_Clean: 0 kB Shared_Clean: 0 kB Shared_Dirty: 0 kB Shared_Dirty: 0 kB Private_Clean: 0 kB Private_Clean: 0 kB Private_Dirty: 28 kB Private_Dirty: 36 kB Referenced: 28 kB Referenced: 36 kB Anonymous: 28 kB Anonymous: 36 kB #INCREASE MEM AnonHugePages: 0 kB AnonHugePages: 0 kB Swap: 0 kB Swap: 0 kB KernelPageSize: 4 kB KernelPageSize: 4 kB MMUPageSize: 4 kB MMUPageSize: 4 kB Locked: 0 kB Locked: 0 kB VmFlags: rd wr mr mw me ac VmFlags: rd wr mr mw me ac
use GDB to dump memory on running process or get the coredump using
gcore -o process
I used gdb on running process to dump the memory to some file.
gdb -p PID dump memory ./dump_outputfile.dump 0x2b3289290000 0x2b3289343000
now, use
strings
command orhexdump -C
to print thedump_outputfile.dump
strings outputfile.dump
You get readable form where you can locate those strings into your source code.
Analyze your source to find the leak.
-
strings command doesn't give me any hint where to look at my source files. for me i just see some garbage strings.– stivAug 19, 2020 at 13:12
I think memleax is exact what you want.
It debugs memory leak of a running process by attaching it, without recompiling program or restarting target process. It's very convenient and suitable for production environment.
It works on GNU/Linux and FreeBSD.
NOTE: I'm the author, any suggestion is welcome.
== EDIT ==
I also wrote another tool libleak, which hooks memory functions by LD_PRELOAD.
There is no need to modify the target program. Although you have to restart the progress with LD_PRELOAD, you can enable/disable the detection during running.
There is much less impact on performance since no signal trap.
Compared with similar tools (such as mtrace), it print the full call-stack at suspicious memory leak point.
-
2I vouch for memleax as a very useful tool to monitor for any obvious leaks. The output summaries are surprisingly effective. Almost like I'd write them if I had the processing power to do it manually. Thanks for this– seheFeb 12, 2017 at 23:54
On Linux, you could enable mtrace in your program, but it is a code change.
On OpenBSD, you could try the malloc stats.
Google's leak checker might also be worth a look, and unlike mtrace you may be able to use LD_PRELOAD
to avoid recompilation.
I think without providing support for allocation monitoring after program start directly in the source code, you're out of luck. Here are two reasons I can think of:
- Heap checkers initialize when the program begins. Some offer the ability to tweak the exact timing, but the environment variables that start them must be set when the program runs. This is because they watch to make sure each allocation has a corresponding deallocation, and they would miss some otherwise.
- Heap checking usually requires elevated privileges, or hooks, to be provided by the operating system. If those hooks aren't provided at the time of program start, the heap checkers cannot leverage them. I don't believe OSes provide these privileges after the program in question is started.
If, however, you're program is running inside a virtual machine, that environment may provide support for monitoring allocations. I know Java has several allocation and garbage collection monitoring tools (like visualVM) that attach to running programs or VMs.
IBM's Purify is probably the oldest and most sophisticated tool of all. It will flag the line number in the code which causes the memory leak.