Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.

I'm in the process of QA/debugging a project on a custom embedded linux system. The problem we're VERY tight on memory, and even a couple megs can cause issues with our software.

What I'm seeing (sometimes) is that when our system crashes (soft crashes), our own software isn't taking up any more memory than usual. But what's weird is that the linux kernel slab has a MUCH higher than normal number of objects in its 32byte slab cache:

size-32           103674 103734     32  113    1 : tunables  120   60    0 : slabdata    918    918      0

This is what the slab looks like under normal circumstances:

size-32             5400   6554     32  113    1 : tunables  120   60    0 : slabdata     58     58      0

Obviously that's a significant difference.

I know there's not much information here, but this is really all the information I have to go on. Is there any way to determine what the slab contains? Or where it was allocated from or anything like that? Or maybe just force it to free its memory?

This is running on an ARM based board with 64mB of ram running a 2.6.30 kernel.

Btw, this is the entirety of what's running on the machine.

10711  8778 root     R     2980  4.9   0  0.0 top
 8745  2740 root     S     6452 10.7   0  0.0 sshd: root@ttyp1
 9411  2740 root     S     6312 10.4   0  0.0 sshd: root@ttyp2
 2740     1 root     S     3704  6.1   0  0.0 /usr/sbin/sshd
 8778  8745 root     S     2984  4.9   0  0.0 -sh
10034  9411 root     S     2984  4.9   0  0.0 -sh
    1     0 root     S     2980  4.9   0  0.0 init
 9432     1 root     S     2848  4.7   0  0.0 runsv ui
 1115     1 root     S <   1576  2.6   0  0.0 udevd --daemon
  989     2 root     SW<      0  0.0   0  0.0 [mmcqd]
  223     2 root     SW<      0  0.0   0  0.0 [kswapd0]
  175     2 root     SW<      0  0.0   0  0.0 [kblockd/0]
  222     2 root     SW       0  0.0   0  0.0 [pdflush]
  221     2 root     SW       0  0.0   0  0.0 [pdflush]
    6     2 root     SW<      0  0.0   0  0.0 [khelper]
    5     2 root     SW<      0  0.0   0  0.0 [events/0]
  186     2 root     SW<      0  0.0   0  0.0 [khubd]
    2     0 root     SW<      0  0.0   0  0.0 [kthreadd]
  220     2 root     SW       0  0.0   0  0.0 [khungtaskd]
    3     2 root     SW<      0  0.0   0  0.0 [ksoftirqd/0]
    4     2 root     SW<      0  0.0   0  0.0 [watchdog/0]
   12     2 root     SW<      0  0.0   0  0.0 [async/mgr]
  189     2 root     SW<      0  0.0   0  0.0 [kseriod]
  194     2 root     SW<      0  0.0   0  0.0 [kmmcd]
  224     2 root     SW<      0  0.0   0  0.0 [aio/0]
  225     2 root     SW<      0  0.0   0  0.0 [unionfs_siod/0]
  226     2 root     SW<      0  0.0   0  0.0 [nfsiod]
  227     2 root     SW<      0  0.0   0  0.0 [crypto/0]
  895     2 root     SW<      0  0.0   0  0.0 [zd1211rw]
  899     2 root     SW<      0  0.0   0  0.0 [mtdblockd]
  972     2 root     SW<      0  0.0   0  0.0 [usbhid_resumer]
  992     2 root     SW<      0  0.0   0  0.0 [rpciod/0]

and it's still growing

size-32           181312 181365     32  113    1 : tunables  120   60    0 : slabdata   1605   1605      0

Update:
This is the current state:

Mem: 38416K used, 21764K free, 0K shrd, 1836K buff, 11548K cached
CPU:  0.0% usr  0.0% sys  0.0% nic  100% idle  0.0% io  0.0% irq  0.0% sirq
Load average: 0.01 0.02 0.05 1/28 11050
  PID  PPID USER     STAT   VSZ %MEM CPU %CPU COMMAND
 8745     1 root     S     6452 10.7   0  0.0 sshd: root@ttyp1
11045     1 root     S     3704  6.1   0  0.0 /usr/sbin/sshd
 8778  8745 root     S     2984  4.9   0  0.0 -sh
    1     0 root     S     2980  4.9   0  0.0 init
11050  8778 root     R     2980  4.9   0  0.0 top
  989     2 root     SW<      0  0.0   0  0.0 [mmcqd]
  223     2 root     SW<      0  0.0   0  0.0 [kswapd0]
  175     2 root     SW<      0  0.0   0  0.0 [kblockd/0]
  222     2 root     SW       0  0.0   0  0.0 [pdflush]
  221     2 root     SW       0  0.0   0  0.0 [pdflush]
    6     2 root     SW<      0  0.0   0  0.0 [khelper]
    5     2 root     SW<      0  0.0   0  0.0 [events/0]
  186     2 root     SW<      0  0.0   0  0.0 [khubd]
    2     0 root     SW<      0  0.0   0  0.0 [kthreadd]
  220     2 root     SW       0  0.0   0  0.0 [khungtaskd]
    3     2 root     SW<      0  0.0   0  0.0 [ksoftirqd/0]
    4     2 root     SW<      0  0.0   0  0.0 [watchdog/0]
   12     2 root     SW<      0  0.0   0  0.0 [async/mgr]
  189     2 root     SW<      0  0.0   0  0.0 [kseriod]
  194     2 root     SW<      0  0.0   0  0.0 [kmmcd]
  224     2 root     SW<      0  0.0   0  0.0 [aio/0]
  225     2 root     SW<      0  0.0   0  0.0 [unionfs_siod/0]
  226     2 root     SW<      0  0.0   0  0.0 [nfsiod]
  227     2 root     SW<      0  0.0   0  0.0 [crypto/0]
  895     2 root     SW<      0  0.0   0  0.0 [zd1211rw]
  899     2 root     SW<      0  0.0   0  0.0 [mtdblockd]
  972     2 root     SW<      0  0.0   0  0.0 [usbhid_resumer]
  992     2 root     SW<      0  0.0   0  0.0 [rpciod/0]


size-32           191069 191083     32  113    1 : tunables  120   60    0 : slabdata   1691   1691      0
share|improve this question
    
Do you have custom kernel code? Can you monitor memory usage regularly and try to associate it with particular syscalls or device events? –  Gilles May 23 '11 at 21:18
    
I have the kernel source, but I'm not 100% sure if it's custom or not, or what's been done to it. I came kind of late onto this project. I can monitor memory usage, but it grows with literally nothing running and there's no interaction with the device. Only what's shown above, which, save for my ssh session, all kernel daemons. I did try killing sshd and restarting it. No change. I unloaded my wifi module (which is the only module on the entire system) and no change. It seems to grow very slowly but consistently. Is there a way I can associate it with syscalls? –  Falmarri May 24 '11 at 0:20
    
You can trace kernel memory allocations. I've never done it so I don't know how practical this is, but it's definitely worth a try. –  Gilles May 24 '11 at 0:27
    
@Gilles: Ideally I'd like to be able to at least get a hint of what's going on without modifying my kernel. I know that's asking a lot though. –  Falmarri May 25 '11 at 19:53
    
Are you sure this is memory in use, not just leftover cache? What does free say? –  vonbrand Jan 25 '13 at 1:59

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.