9

Output of free showed that my applications only use 2GB, swap was not used at all. Yet my applications were being killed (Skype, Firefox, Thunderbird). I watched it on htop and it looked like they allocated few megabytes and got killed. I had to reboot to "fix" the problem.

My question is why and more importantly how to fix it when this happens again (I mean other than restart)? (EDIT3)

It was very annoying, I lost quite a long post (Firefox was killed without any notice) and other things. Is it normal on Linux, killing applications b/c OOM on 80% free memory and with no attempt to use swap at all?

EDIT: Distribution is Kubuntu 16.04.

EDIT2: Log (small fragment, b/c of post length limit):

[ 2687.946164] Xorg invoked oom-killer: gfp_mask=0x24040c0, order=3, oom_score_adj=0
[ 2687.946167] Xorg cpuset=/ mems_allowed=0
[ 2687.946171] CPU: 3 PID: 2109 Comm: Xorg Tainted: P           OE   4.4.0-59-generic #80-Ubuntu
[ 2687.946172] Hardware name: MSI MS-7850/B85-G41 PC Mate(MS-7850), BIOS V2.9 03/30/2015
[ 2687.946174]  0000000000000286 00000000828bbbc2 ffff880409ad38d8 ffffffff813f7583
[ 2687.946176]  ffff880409ad3ab0 ffff880392c09c00 ffff880409ad3948 ffffffff8120ad5e
[ 2687.946177]  0000000000000015 0000000000000000 ffff8802e4f47180 ffff8800033c2a00
[ 2687.946179] Call Trace:
...
[ 2687.946394] Mem-Info:
[ 2687.946396] active_anon:720752 inactive_anon:279235 isolated_anon:0
[ 2687.946396]  active_file:729283 inactive_file:2048755 isolated_file:0
[ 2687.946396]  unevictable:24 dirty:34175 writeback:29190 unstable:0
[ 2687.946396]  slab_reclaimable:107795 slab_unreclaimable:37549
[ 2687.946396]  mapped:269800 shmem:51809 pagetables:15581 bounce:0
[ 2687.946396]  free:38642 free_pcp:46 free_cma:0
[ 2687.946398] Node 0 DMA free:15900kB min:64kB low:80kB high:96kB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15984kB managed:15900kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:0kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes
[ 2687.946401] lowmem_reserve[]: 0 3401 15912 15912 15912
[ 2687.946403] Node 0 DMA32 free:69016kB min:14432kB low:18040kB high:21648kB active_anon:393040kB inactive_anon:463048kB active_file:635316kB inactive_file:1758444kB unevictable:80kB isolated(anon):0kB isolated(file):0kB present:3613072kB managed:3532452kB mlocked:80kB dirty:26300kB writeback:23864kB mapped:226516kB shmem:47828kB slab_reclaimable:91324kB slab_unreclaimable:56464kB kernel_stack:2704kB pagetables:13724kB unstable:0kB bounce:0kB free_pcp:0kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[ 2687.946406] lowmem_reserve[]: 0 0 12510 12510 12510
[ 2687.946408] Node 0 Normal free:69652kB min:53084kB low:66352kB high:79624kB active_anon:2489968kB inactive_anon:653892kB active_file:2281816kB inactive_file:6436576kB unevictable:16kB isolated(anon):0kB isolated(file):0kB present:13074432kB managed:12810984kB mlocked:16kB dirty:110400kB writeback:92896kB mapped:852684kB shmem:159408kB slab_reclaimable:339856kB slab_unreclaimable:93732kB kernel_stack:11072kB pagetables:48600kB unstable:0kB bounce:0kB free_pcp:172kB local_pcp:0kB free_cma:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? no
[ 2687.946411] lowmem_reserve[]: 0 0 0 0 0
[ 2687.946412] Node 0 DMA: 1*4kB (U) 1*8kB (U) 1*16kB (U) 0*32kB 2*64kB (U) 1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (M) 3*4096kB (M) = 15900kB
[ 2687.946419] Node 0 DMA32: 6415*4kB (UME) 2655*8kB (UME) 1398*16kB (UME) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 69268kB
[ 2687.946424] Node 0 Normal: 10812*4kB (ME) 2620*8kB (UM) 72*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (H) = 69456kB
[ 2687.946439] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
[ 2687.946440] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
[ 2687.946441] 2829837 total pagecache pages
[ 2687.946442] 0 pages in swap cache
[ 2687.946443] Swap cache stats: add 0, delete 0, find 0/0
[ 2687.946443] Free swap  = 16699388kB
[ 2687.946444] Total swap = 16699388kB
[ 2687.946444] 4175872 pages RAM
[ 2687.946445] 0 pages HighMem/MovableOnly
[ 2687.946445] 86038 pages reserved
[ 2687.946446] 0 pages cma reserved
[ 2687.946446] 0 pages hwpoisoned
[ 2687.946447] [ pid ]   uid  tgid total_vm      rss nr_ptes nr_pmds swapents oom_score_adj name
[ 2687.946453] [  445]     0   445     8850     1198      20       3        0             0 systemd-journal
[ 2687.946454] [  478]     0   478    25742      381      17       3        0             0 lvmetad
[ 2687.946456] [  492]     0   492    11323     1100      23       3        0         -1000 systemd-udevd
[ 2687.946458] [ 1070]   100  1070    25596      640      21       3        0             0 systemd-timesyn
[ 2687.946459] [ 1112]     0  1112     1100      333       8       3        0             0 acpid
[ 2687.946461] [ 1116]     0  1116    41637     2192      36       3        0             0 thermald
[ 2687.946462] [ 1118]     0  1118     7280      757      19       3        0             0 cron
[ 2687.946463] [ 1120]   108  1120    91142     3432      78       3        0             0 whoopsie
[ 2687.946465] [ 1122]   106  1122    10949     1230      27       3        0          -900 dbus-daemon
[ 2687.946466] [ 1130]     0  1130     3964      622      14       3        0             0 anacron
[ 2687.946467] [ 1132]     0  1132   112330     4027      71       3        0             0 NetworkManager
[ 2687.946468] [ 1133]     0  1133    53957     4872      29       5        0             0 snapd
[ 2687.946470] [ 1165]     0  1165    84379     2173      66       3        0             0 ModemManager
[ 2687.946471] [ 1167]   104  1167    64099      871      27       3        0             0 rsyslogd
[ 2687.946472] [ 1168]   122  1168    33544     4711      53       3        0             0 freshclam
[ 2687.946473] [ 1170]   110  1170    11229      911      27       3        0             0 avahi-daemon
[ 2687.946475] [ 1174]     0  1174     7157      789      18       3        0             0 systemd-logind
[ 2687.946476] [ 1183]     0  1183    69059     2174      38       3        0             0 accounts-daemon
[ 2687.946477] [ 1224]   110  1224    11197       85      26       3        0             0 avahi-daemon
[ 2687.946479] [ 1270]     0  1270    70587     2236      41       3        0             0 polkitd
[ 2687.946480] [ 1299]   126  1299    73695     6057      71       4        0          -900 postgres
[ 2687.946481] [ 1302]   126  1302    73695     1019      58       4        0             0 postgres
[ 2687.946482] [ 1303]   126  1303    73695     1425      59       4        0             0 postgres
[ 2687.946483] [ 1304]   126  1304    73695     1019      57       4        0             0 postgres
[ 2687.946485] [ 1305]   126  1305    73802     1637      62       4        0             0 postgres
[ 2687.946486] [ 1306]   126  1306    37475      830      55       4        0             0 postgres
[ 2687.946487] [ 1680]     0  1680     4030      892      13       3        0             0 dhclient
[ 2687.946489] [ 1692] 65534  1692    13245     1056      31       3        0             0 dnsmasq
[ 2687.946490] [ 1971]     0  1971     4868      572      14       3        0             0 irqbalance
[ 2687.946491] [ 2027]   127  2027   940075   118111     325       7        0             0 java
[ 2687.946493] [ 2049]     0  2049    26294     3360      41       3        0             0 sddm
[ 2687.946494] [ 2099]     0  2099     5323      878      16       3        0             0 mysqld_safe
[ 2687.946495] [ 2107]     0  2107    27240     1164      53       3        0             0 sendmail-mta
[ 2687.946496] [ 2109]     0  2109    84298    54063     166       3        0             0 Xorg
[ 2687.946498] [ 2254]   129  2254   151112    24593      89       3        0             0 mysqld
[ 2687.946499] [ 2255]     0  2255     6551      344      18       3        0             0 logger
[ 2687.946500] [ 2278]   128  2278    54060     3442      52       4        0             0 murmurd
[ 2687.946502] [ 2299]     0  2299    60029     1474     113       4        0             0 nmbd
[ 2687.946503] [ 2312]   120  2312     4275      408      14       3        0             0 nvidia-persiste
[ 2687.946504] [ 2324]     0  2324    71846     2203     135       3        0             0 winbindd
[ 2687.946506] [ 2325]     0  2325    71844     2273     136       3        0             0 winbindd
[ 2687.946507] [ 2331]     0  2331    84508     3947     163       3        0             0 smbd
[ 2687.946508] [ 2332]     0  2332    82479     1147     152       3        0             0 smbd
[ 2687.946509] [ 2335]     0  2335    71845     1632     136       3        0             0 winbindd
[ 2687.946511] [ 2336]     0  2336    71845     1031     134       3        0             0 winbindd
[ 2687.946512] [ 2338]     0  2338    33686      884      60       3        0             0 veracrypt
[ 2687.946513] [ 2339]     0  2339    78817      875      69       3        0             0 veracrypt
[ 2687.946514] [ 2347]     0  2347    84508     1814     156       3        0             0 smbd
[ 2687.946516] [ 2478]     0  2478    31094     2925      52       3        0             0 sddm-helper
[ 2687.946517] [ 2490]  1000  2490    11345     1227      25       4        0             0 systemd
[ 2687.946518] [ 2491]  1000  2491    15954      614      34       3        0             0 (sd-pam)
[ 2687.946520] [ 2500]  1000  2500     1127      403       8       3        0             0 startkde
[ 2687.946521] [ 2578]     0  2578     7534     4948      18       3        0             0 mount.ntfs
[ 2687.946522] [ 2583]  1000  2583    41663      137      18       3        0             0 gpg-agent
[ 2687.946523] [ 2586]  1000  2586     2785       80       9       3        0             0 ssh-agent
[ 2687.946525] [ 2590]  1000  2590    10900      643      23       3        0             0 dbus-launch
[ 2687.946526] [ 2596]  1000  2596    11038     1108      24       3        0             0 dbus-daemon
[ 2687.946527] [ 2642]     0  2642    33690      970      62       3        0             0 veracrypt
[ 2687.946528] [ 2643]     0  2643    78821      876      70       4        0             0 veracrypt
[ 2687.946530] [ 2704]  1000  2704     1575       24       8       3        0             0 start_kdeinit
[ 2687.946531] [ 2706]  1000  2706    66992     5454     113       3        0             0 kdeinit5
[ 2687.946533] [ 2710]  1000  2710   104611     7283     150       3        0             0 klauncher
[ 2687.946534] [ 2712]     0  2712    24833    22247      52       3        0             0 mount.ntfs
[ 2687.946535] [ 2718]  1000  2718   377382    54973     375       4        0             0 kded5
[ 2687.946536] [ 2732]  1000  2732   101975     7434     147       4        0             0 kaccess
[ 2687.946538] [ 2743]  1000  2743    17577     1667      29       3        0             0 kwrapper5
[ 2687.946539] [ 2746]  1000  2746   128035     9524     166       3        0             0 ksmserver
[ 2687.946540] [ 2748]     0  2748    33690      986      61       3        0             0 veracrypt
[ 2687.946542] [ 2749]     0  2749    78821      879      69       3        0             0 veracrypt
[ 2687.946543] [ 2762]  1000  2762    99939     8388     145       3        0             0 kglobalaccel5
[ 2687.946544] [ 2765]     0  2765    86884     2600      54       4        0             0 upowerd
[ 2687.946545] [ 2768]  1000  2768   194905     8632     158       3        0             0 kactivitymanage
[ 2687.946547] [ 2785]  1000  2785    44633     1163      23       3        0             0 dconf-service
[ 2687.946548] [ 2792]  1000  2792   796368    30269     277       6        0             0 kwin_x11
[ 2687.946549] [ 2796]  1000  2796  1371562     6937      87       5        0             0 baloo_file
[ 2687.946551] [ 2802]  1000  2802  1679410    43163     355       8        0             0 krunner
[ 2687.946552] [ 2818]     0  2818   108587     2972      48       3        0             0 udisksd
[ 2687.946554] [ 2863]  1000  2863  1146275    97958     559       7        0             0 plasmashell
[ 2687.946556] [ 2879]     0  2879     3199      637      10       3        0             0 mount.ntfs
[ 2687.946557] [ 2882]  1000  2882   107872     3747      61       3        0             0 mission-control
[ 2687.946559] [ 2894]     0  2894     4013      419      13       3        0             0 agetty
[ 2687.946560] [ 2898]  1000  2898   140656     8810     162       4        0             0 polkit-kde-auth
[ 2687.946561] [ 2905]  1000  2905    63342     4444      80       3        0             0 kscreen_backend
[ 2687.946562] [ 2914]  1000  2914   100125     7779     143       3        0             0 xembedsniproxy
[ 2687.946564] [ 2919]  1000  2919   114254    11562     173       3        0             0 charmtimetracke
[ 2687.946565] [ 2926]  1000  2926   131566     3645     101       4        0             0 pulseaudio
[ 2687.946567] [ 2927]   116  2927    45886      752      25       4        0             0 rtkit-daemon
[ 2687.946568] [ 2928]  1000  2928    75967     9835     135       3        0             0 qstardict
[ 2687.946570] [ 2932]  1000  2932   440694    28716     359       5        0             0 RetroShare06
[ 2687.946571] [ 2945]  1000  2945    68633     1616      36       4        0             0 gvfsd
[ 2687.946572] [ 2953]  1000  2953    85331     1850      32       4        0             0 gvfsd-fuse
[ 2687.946573] [ 2972]  1000  2972   779950    43029     349       6        0             0 dropbox
[ 2687.946574] [ 2978]  1000  2978   331520    24592     233      19        0             0 skypeforlinux
[ 2687.946576] [ 2984]  1000  2984   141037    18264     189       3        0             0 yakuake
[ 2687.946577] [ 2985]  1000  2985    84453     1369      32       3        0             0 at-spi-bus-laun
[ 2687.946578] [ 2988]  1000  2988    51367     1877      33       4        0             0 gnome-keyring-d
[ 2687.946579] [ 2997]  1000  2997   178857    12805     221       3        0             0 korgac
[ 2687.946580] [ 2999]  1000  2999    10757      972      27       4        0             0 dbus-daemon
[ 2687.946582] [ 3002]  1000  3002    51741     1317      36       3        0             0 at-spi2-registr
[ 2687.946583] [ 3015]  1000  3015   626351    12095     116       5        0             0 copyq
[ 2687.946584] [ 3037]  1000  3037    78126     7392     125       4        0             0 skypeforlinux
[ 2687.946585] [ 3047]  1000  3047    37586     2507      57       3        0             0 obexd
[ 2687.946587] [ 3059]  1000  3059    25344      999      51       3        0             0 gconf-helper
[ 2687.946588] [ 3069]  1000  3069    15384     1472      35       3        0             0 gconfd-2
[ 2687.946589] [ 3079]   107  3079     7067      328      20       3        0             0 uuidd
[ 2687.946590] [ 3085]  1000  3085    87332     7609     126       3        0             0 copyq
[ 2687.946592] [ 3100]  1000  3100    43529     3916      71       4        0             0 kdeinit4
[ 2687.946593] [ 3103]  1000  3103    49619     4728      80       4        0             0 klauncher
[ 2687.946595] [ 3117]  1000  3117    69575     6869     116       4        0             0 kded4
[ 2687.946596] [ 3123]  1000  3123     2543      588      10       3        0             0 ksysguardd
[ 2687.946597] [ 3125]  1000  3125    11789     1615      28       3        0             0 zsh
[ 2687.946598] [ 3129]  1000  3129    97321     8939     141       3        0             0 knotify4
[ 2687.946600] [ 3175]  1000  3175   121189    26207     204       6        0             0 skypeforlinux
[ 2687.946601] [ 3274]  1000  3274   313212    42209     324      86        0           300 skypeforlinux
[ 2687.946603] [ 3379]  1000  3379   123103     9279     157       3        0             0 kdeconnectd
[ 2687.946604] [ 3387]  1000  3387    68391     6958     116       3        0             0 kuiserver
[ 2687.946605] [ 3507]  1000  3507   342526    29181     257       4        0             0 doublecmd
[ 2687.946606] [ 3521]  1000  3521   623171   332899    1038       6        0             0 firefox
[ 2687.946607] [ 3544]  1000  3544   343564    91599     443       5        0             0 thunderbird
[ 2687.946609] [ 3800]  1000  3800    81605     2254      45       4        0             0 gvfsd-http
[ 2687.946610] [ 4353]     0  4353     1127      195       8       3        0             0 sh
[ 2687.946611] [ 4354]     0  4354     1092      367       8       3        0             0 run-parts
[ 2687.946612] [ 4510]     0  4510    68705     2415      68       3        0             0 cups-browsed
[ 2687.946613] [ 5001]     0  5001     3160      736      12       3        0             0 sbackup
[ 2687.946614] [ 5002]     0  5002   101153    19913      97       3        0             0 sbackup
[ 2687.946615] [ 5038]     0  5038    10936      549      24       3        0             0 dbus-launch
[ 2687.946617] [ 5039]     0  5039    10692      644      26       3        0             0 dbus-daemon
[ 2687.946618] [ 5041]     0  5041    15355     1394      34       3        0             0 gconfd-2
[ 2687.946619] [ 5046]     0  5046    10692      773      25       3        0             0 dbus-daemon
[ 2687.946620] [ 5047] 65534  5047    26365     4559      54       4        0             0 sbackup-dbusser
[ 2687.946622] [ 5055]  1000  5055   121399     8962     119       4        0             0 sbackup-indicat
[ 2687.946623] [ 5080]     0  5080    68633     1553      34       3        0             0 gvfsd
[ 2687.946624] [ 5085]     0  5085   101715     1820      33       4        0             0 gvfsd-fuse
[ 2687.946625] [ 5093]     0  5093    46523     1340      26       3        0             0 gvfsd-metadata
[ 2687.946626] [ 5865]     0  5865    80226    74366     162       3        0             0 tar
[ 2687.946627] [ 5867]     0  5867     1192      427       8       3        0             0 gzip
[ 2687.946628] [ 6688]  1000  6688     3183      805      11       3        0             0 bash
[ 2687.946630] [ 6697]  1000  6697     3235      857      12       3        0             0 bash
[ 2687.946631] [ 6698]  1000  6698     1853      164       9       3        0             0 tee
[ 2687.946632] [ 7338]  1000  7338    46611     1396      28       3        0             0 gvfsd-metadata
[ 2687.946634] [ 8551]   126  8551    73917     2014      63       4        0             0 postgres
[ 2687.946635] [ 8552]   126  8552    73917     2022      63       4        0             0 postgres
[ 2687.946636] [ 8554]   126  8554    73917     2051      63       4        0             0 postgres
[ 2687.946637] [ 8556]   126  8556    73917     2051      63       4        0             0 postgres
[ 2687.946639] [ 8559]   126  8559    73917     2051      63       4        0             0 postgres
[ 2687.946640] [ 8560]   126  8560    73917     2051      63       4        0             0 postgres
[ 2687.946641] [ 8562]   126  8562    73917     2051      63       4        0             0 postgres
[ 2687.946642] [ 8643]   126  8643    73917     2051      63       4        0             0 postgres
[ 2687.946644] [ 8645]   126  8645    73917     2051      63       4        0             0 postgres
[ 2687.946645] [ 8648]   126  8648    73917     2051      63       4        0             0 postgres
[ 2687.946646] [ 8649]   126  8649    73917     2051      63       4        0             0 postgres
[ 2687.946647] [ 8652]   126  8652    73917     2051      63       4        0             0 postgres
[ 2687.946648] [ 8653]   126  8653    73917     2051      63       4        0             0 postgres
[ 2687.946649] [ 8656]   126  8656    73917     2038      63       4        0             0 postgres
[ 2687.946650] [ 8657]   126  8657    73917     2036      63       4        0             0 postgres
[ 2687.946652] [ 8659]   126  8659    73917     2051      63       4        0             0 postgres
[ 2687.946653] [ 8661]   126  8661    73917     2051      63       4        0             0 postgres
[ 2687.946654] [ 8663]   126  8663    73917     2051      63       4        0             0 postgres
[ 2687.946655] [ 8665]   126  8665    73917     2051      63       4        0             0 postgres
[ 2687.946656] [ 8667]   126  8667    73917     2051      63       4        0             0 postgres
[ 2687.946658] [ 9415]  1000  9415   135556    80527     253       2        0             0 steam
[ 2687.946659] [ 9417]  1000  9417    33897    13343      58       2        0             0 steam
[ 2687.946660] [ 9418]  1000  9418     1127      176       8       3        0             0 sh
[ 2687.946661] [ 9420]  1000  9420     3164      767      12       3        0             0 steamwebhelper.
[ 2687.946663] [ 9422]  1000  9422   151458    19771     193       5        0             0 steamwebhelper
[ 2687.946664] [ 9424]  1000  9424    78943     9566     131       4        0             0 steamwebhelper
[ 2687.946666] [ 9542]  1000  9542   232518    13239     162      25        0           300 steamwebhelper
[ 2687.946667] [ 9606]  1000  9606   233909    14334     165      26        0           300 steamwebhelper
[ 2687.946668] [ 9734]   111  9734    76055     3464      50       3        0             0 colord
[ 2687.946669] [10387]     0 10387    23345     1859      46       3        0             0 cupsd
[ 2687.946671] [10389]     7 10389    20311     1396      41       3        0             0 dbus
[ 2687.946672] [10390]     7 10390    20311     1441      43       3        0             0 dbus
[ 2687.946673] Out of memory: Kill process 3274 (skypeforlinux) score 305 or sacrifice child
[ 2687.946697] Killed process 3274 (skypeforlinux) total-vm:1252848kB, anon-rss:103412kB, file-rss:65424kB

I suspect that it has to do something with not freeing io cache, unfortunately I did not save output of free.

EDIT3: I thought that question in bold was clearly marked, but some people failed to read and started marking this whole post as "put on hold as unclear what you're asking". I have provided all additional requested info and just to be sure edited title to be a question to avoid situation when someone ignores question text and marks it straight away as "unclear what question is"...

EDIT4: Happened again, definitely related to high IO (reading from HDD). Output of free:

              total        used        free      shared  buff/cache   available
Mem:          15975        3535         294         151       12145       11867
Swap:         16307           1       16306
11
  • 3
    You will need to provide the full output of the OOM killer message. This cannot be answered without that.
    – phemmer
    Jan 15 '17 at 7:54
  • 1
    Strange that swap is not used...could you please add the output of cat /proc/sys/vm/swappiness.
    – Thomas
    Jan 15 '17 at 13:42
  • 1
    Well, your memory is extremely fragmented. your DMA32 has 0 pages >= order=3, and normal only has an order=12 chunk (which can be broken into smaller chunks, but it still means you're low). However the gfp_mask indicates that it didn't specifically ask for the DMA32 zone, so it should be able to pull from normal.
    – phemmer
    Jan 15 '17 at 17:51
  • 4
    Why is this closed "unclear what you're asking"? There is a very clear question asked, and there are no outstanding requests for more information. The question has the oom killer message provided, which should be more than enough to figure it out.
    – phemmer
    Jan 18 '17 at 13:26
  • 2
    I don't get it why are people voting to close this (are they even reading it?). Box about "put on hold" does not apply to this question - question is clearly stated (in bold font and now even in title) and I answered all additional info requests. I am starting to think stackexchange is a bad site for asking questions and those rumors about being filled with trolls might be right :-/.
    – monnef
    Jan 18 '17 at 14:14
3

To your questions:
Why: You are probably using Kernel 4.4.0-59. If so you are affected by a OOM bug: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1655842.
How to fix it when this happens again: Downgrade your kernel to eg. 4.4.0-57.

1
  • Yes, I am using that (default) version of kernel (4.4.0-59-generic). I'll try downgrading, thank you for suggestion.
    – monnef
    Jan 19 '17 at 14:03
0

The cause of your system killing a process is memory fragmentation. However, you didn't have 14 GB of free RAM but around 70 MB while this happened.

The interesting lines in the log are these:

[ 2687.946164] Xorg invoked oom-killer: gfp_mask=0x24040c0, order=3, oom_score_adj=0

[ 2687.946424] Node 0 Normal: 10812*4kB (ME) 2620*8kB (UM) 72*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 1*4096kB (H) = 69456kB

The first line with order=3 tells that that allocation size was 32 KB (the value is computed page_size * 2^order or 4096*2*2*2 = 32768 bytes.

The second line tells that the system had 0 pages free with that size. The biggest memory fragments that were free were 16 KB blocks and you had only 72 of those, too.

The meaning of gfp_mask has to be decoded using the values in https://github.com/torvalds/linux/blob/master/include/linux/gfp.h

However, it's my understanding that because this request had ___GFP_RETRY_MAYFAIL set the kernel should have tried to reclaim or compact the memory and retry instead of triggering OOM Killer. That might be a kernel bug in the kernel version that was in use while this happened. Or your kernel version was compiled with memory compaction disabled.

Creating a cron job that periodically runs echo 1 > /proc/sys/vm/compact_memory as root as a workaround might be able to avoid this problem if your kernel cannot compact the memory automatically. However, that will cause some additional CPU usage so it shouldn't be run if you upgrade the kernel instead.

1
  • Also note that you would need to configure kernel to swap pages to reclaim memory blocks to avoid OOM Killer if that's what you want by running echo 7 > /proc/sys/vm/zone_reclaim_mode. That will reduce system throughput in some cases but allow you to run closer to OOM without OOM Killer being triggered. Aug 22 at 20:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.