Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

After starting VirtualBox, the computer became sluggish and then hung completely due to OOM. Usually, OOM should be starting killing processes in order to free up some space, but this did not happen (this was the second time that I experienced this).

I had some unsaved important work in a text editor, so I was hoping to find it back in the system RAM after killing all processes in the current console using SysRq + K. The machine in question is a laptop with 8 GiB RAM running Linux x86_64 3.7.5 with a SSD as target disk.

My first attempt was dd if=/dev/mem of=memory, but this failed after reading 1MiB of data. Next, I tried dd if=/dev/fmem of=memory bs=1M, but this stopped after reading 3010461696 bytes (exactly 2871 MiB). After looking at /proc/mtrr (shown below), I decided to try adding skip=4096. This ultimately slowed down, reading at a speed of only 3 MiB/sec, so I interrupted it (yielding a file of 5.8 GiB). (at least the last 100 MiB of the file contains FFs)

reg01: base=0x000000000 (    0MB), size= 2048MB, count=1: write-back
reg02: base=0x080000000 ( 2048MB), size= 1024MB, count=1: write-back
reg03: base=0x100000000 ( 4096MB), size= 4096MB, count=1: write-back
reg04: base=0x200000000 ( 8192MB), size= 1024MB, count=1: write-back
reg05: base=0x23c000000 ( 9152MB), size=   64MB, count=1: uncachable
reg06: base=0x0b4000000 ( 2880MB), size=   64MB, count=1: uncachable
reg07: base=0x0b8000000 ( 2944MB), size=  128MB, count=1: uncachable

I could not find the data I had open for some hours in the text editor, so I believe I have skipped some memory while doing a dump. So, given my goal (recovery of data from userspace programs), what is the most efficient method to dump the system memory to a file? What are some points that must be considered while doing such a dump?

share|improve this question
    
Have you tried /proc/kmem too? It's not much worth because it changes while you're copying it tho. –  ott-- Feb 24 '13 at 22:16
    
@ott-- CONFIG_DEVKMEM is disabled, looking in the source code it seem to allow unrestricted access, but I am still not convinced that this is the best way to do it (IO mem access?) –  Lekensteyn Feb 24 '13 at 22:40
2  
Maybe you did get the memory of the editor but you didn't recognize it. Reconstructing data structures from a memory dump can be hard. The first thing you'll need to do is reconstruct the memory mapping from the kernel data structures (there are probably existing forensic tools for that) so as to get the virtual memory of the process you're interested in (which is likely to be spread around many disjoint 4kB pages in physical memory). Then the text may not be in one consecutive blob and may use UCS4 or other representations and may store lines or other blocks in separate chunks. –  Gilles Feb 24 '13 at 23:00
1  
@Gilles +1, once the process is killed I would expect kernel to free the task descriptors -> forget all about its address space mapping. As for the data representation, it can easily be a tree (with enough luck allocated by JVM :)). –  peterph Feb 25 '13 at 0:20
    
So you are going to dig through gigabytes of data looking for a few kb of text that may not even be there? Needle, meet haystack. In the time you spent on this you could have just retyped the text. If there was that much in the first place, then you should make sure your text editor is configured to periodically save a backup so you don't lose much if it crashes. –  psusi Feb 25 '13 at 2:54
show 2 more comments

2 Answers

Check this project: foriana

Foriana is (FOrensic Ram Image ANAlyzer)

input: dump of (physical) RAM output: various information

Version 1.0 can list processes and modules from memory dump of i386/x86_64/arm linux/bsd kernels, and provide option for reading linear memory from dumps.

There is a kernel module fmem:

Fmem is kernel driver, that creates /dev/fmem device. /dev/fmem behave in same way that /dev/mem (direct access to physical memory), but does not have limits that /dev/mem have. It is possible to dump whole physical memory through /dev/fmem.

I have use it, compile pretty easy.

share|improve this answer
    
The asker did try /dev/fmem. –  Tobu Jul 14 '13 at 6:52
add comment

You might want to use ddrescue or similar program, that can skip inaccessible data. dd conv=noerror might be helpful too. Also check this question on superuser.

More importantly though, if you got yourself into an OOM situation the sluggishness was very likely caused by kernel swapping out pages from anything else than the requesting application. Hence if you want your data, check the swap instead of /dev/mem - chances are it will be there. Similarly, if the OOM killer doesn't kick in and you kill processes by hand, once e.g. your editor gets killed first the memory hungry process might still happen to get some time to grab those pages.

As mentioned by Gilles in comment, the data can easily be in some special structure, so you won't be able to find them that easily even if you manage to reconstruct killed process's address space mappings and have enough luck to find all of the needed pages still intact.

share|improve this answer
1  
I have seen that SU answer before, that is how I found fmem. ddrescue is not going to help me since 256 pages (1 MiB) is a hard-coded limit. I would expect to get into an OOM condition, but the OOM killer did not kick in (pastebin.com/DvYTCcRK). One week ago, I had the same issue (still Linux 3.7.5, I have not rebooted, only suspended to ram). There is no swap file/partition since I have a SSD. (swappiness = 60 (default)). –  Lekensteyn Feb 24 '13 at 22:53
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.