1

I run a debian Webserver with nextcloud. Every night a backup job (storeBackup) is scheduled and it only succeeds in very few cases. But every night the journal is full of OOM Errors (see below).

The machine has 16G RAM, therefore I am wondering, what is so much RAM consuming.

Output of journalctl -p err -e for one night:

Jun 21 03:28:01 balkonien kernel: Out of memory: Kill process 4723 (clamd) score 19 or sacrifice child
Jun 21 03:28:01 balkonien kernel: Killed process 4723 (clamd) total-vm:760960kB, anon-rss:49044kB, file-rss:6504kB, shmem-rss:0kB
Jun 21 03:35:00 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:35:00 balkonien kernel: Killed process 22139 (loolkit) total-vm:1488072kB, anon-rss:53768kB, file-rss:0kB, shmem-rss:0kB
Jun 21 03:38:11 balkonien kernel: Out of memory: Kill process 8246 (loolkit) score 13 or sacrifice child
Jun 21 03:38:11 balkonien kernel: Killed process 8246 (loolkit) total-vm:1488072kB, anon-rss:58388kB, file-rss:25816kB, shmem-rss:0kB
Jun 21 03:40:49 balkonien kernel: Out of memory: Kill process 10165 (loolkit) score 13 or sacrifice child
Jun 21 03:40:49 balkonien kernel: Killed process 10165 (loolkit) total-vm:1488072kB, anon-rss:58336kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:38 balkonien kernel: Out of memory: Kill process 12575 (loolkit) score 13 or sacrifice child
Jun 21 03:47:38 balkonien kernel: Killed process 12575 (loolkit) total-vm:1488072kB, anon-rss:58212kB, file-rss:26224kB, shmem-rss:0kB
Jun 21 03:47:39 balkonien kernel: Out of memory: Kill process 22111 (forkit) score 12 or sacrifice child
Jun 21 03:47:39 balkonien kernel: Killed process 22111 (forkit) total-vm:1487792kB, anon-rss:53880kB, file-rss:2080kB, shmem-rss:0kB
Jun 21 03:48:45 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 17 or sacrifice child
Jun 21 03:48:45 balkonien kernel: Killed process 16783 (loolkit) total-vm:1488068kB, anon-rss:350976kB, file-rss:27500kB, shmem-rss:0kB
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child
Jun 21 04:07:00 balkonien kernel: Killed process 16801 (loolkit) total-vm:1488068kB, anon-rss:349752kB, file-rss:7536kB, shmem-rss:0kB                                                                                                                                         
Jun 21 04:07:00 balkonien kernel: Out of memory: Kill process 16652 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 04:07:00 balkonien kernel: Killed process 16652 (forkit) total-vm:1487720kB, anon-rss:349536kB, file-rss:18508kB, shmem-rss:0kB                                                                                                                                         
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 04:49:00 balkonien kernel: Killed process 27082 (loolkit) total-vm:1488068kB, anon-rss:102080kB, file-rss:428kB, shmem-rss:0kB                                                                                                                                          
Jun 21 04:49:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 04:49:00 balkonien kernel: Killed process 27088 (loolkit) total-vm:1488068kB, anon-rss:101984kB, file-rss:416kB, shmem-rss:0kB                                                                                                                                          
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 12165 (loolkit) score 12 or sacrifice child                                                                                                                                                                      
Jun 21 04:56:00 balkonien kernel: Killed process 12165 (loolkit) total-vm:1488068kB, anon-rss:100852kB, file-rss:7140kB, shmem-rss:0kB                                                                                                                                         
Jun 21 04:56:00 balkonien kernel: Out of memory: Kill process 27040 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 04:56:00 balkonien kernel: Killed process 27040 (forkit) total-vm:1487720kB, anon-rss:99072kB, file-rss:60kB, shmem-rss:0kB                                                                                                                                             
Jun 21 05:07:01 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 05:07:01 balkonien kernel: Killed process 13170 (loolkit) total-vm:1488068kB, anon-rss:348584kB, file-rss:840kB, shmem-rss:0kB                                                                                                                                          
Jun 21 05:09:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 05:09:00 balkonien kernel: Killed process 13174 (loolkit) total-vm:1488068kB, anon-rss:340688kB, file-rss:0kB, shmem-rss:0kB                                                                                                                                            
Jun 21 05:09:43 balkonien kernel: Out of memory: Kill process 14902 (loolkit) score 13 or sacrifice child                                                                                                                                                                      
Jun 21 05:09:43 balkonien kernel: Killed process 14902 (loolkit) total-vm:1488068kB, anon-rss:340476kB, file-rss:26308kB, shmem-rss:0kB                                                                                                                                        
Jun 21 05:54:52 balkonien kernel: Out of memory: Kill process 15401 (loolkit) score 12 or sacrifice child                                                                                                                                                                      
Jun 21 05:54:52 balkonien kernel: Killed process 15401 (loolkit) total-vm:1488068kB, anon-rss:295604kB, file-rss:1080kB, shmem-rss:0kB                                                                                                                                         
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 15519 (loolkit) score 13 or sacrifice child                                                                                                                                                                      
Jun 21 05:56:00 balkonien kernel: Killed process 15519 (loolkit) total-vm:1488068kB, anon-rss:292004kB, file-rss:25992kB, shmem-rss:0kB                                                                                                                                        
Jun 21 05:56:00 balkonien kernel: Out of memory: Kill process 13137 (forkit) score 12 or sacrifice child                                                                                                                                                                       
Jun 21 05:56:00 balkonien kernel: Killed process 13137 (forkit) total-vm:1487720kB, anon-rss:291656kB, file-rss:1272kB, shmem-rss:0kB                                                                                                                                          
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child                                                                                                                                                                         
Jun 21 05:56:07 balkonien kernel: Killed process 27688 (perl) total-vm:34744kB, anon-rss:4820kB, file-rss:3848kB, shmem-rss:0kB                                                                                                                                                
Jun 21 05:56:07 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child                                                                                                                                                                         
Jun 21 05:56:07 balkonien kernel: Killed process 27703 (perl) total-vm:34728kB, anon-rss:5364kB, file-rss:3812kB, shmem-rss:0kB                                                                                                                                                
Jun 21 05:56:15 balkonien kernel: Out of memory: Kill process 16287 (forkit) score 13 or sacrifice child                                                                                                                                                                       
Jun 21 05:56:15 balkonien kernel: Killed process 16287 (forkit) total-vm:1381924kB, anon-rss:248144kB, file-rss:129808kB, shmem-rss:0kB                                                                                                                                        
Jun 21 05:58:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child                                                                                                                                                                       
Jun 21 05:58:18 balkonien kernel: Killed process 16655 (loolkit) total-vm:1488068kB, anon-rss:348072kB, file-rss:25648kB, shmem-rss:0kB
Jun 21 05:59:18 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 16 or sacrifice child
Jun 21 05:59:18 balkonien kernel: Killed process 16675 (loolkit) total-vm:1488068kB, anon-rss:345868kB, file-rss:24916kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 17546 (loolkit) total-vm:1488068kB, anon-rss:342780kB, file-rss:3176kB, shmem-rss:0kB
Jun 21 06:07:00 balkonien kernel: Out of memory: Kill process 16503 (forkit) score 12 or sacrifice child
Jun 21 06:07:00 balkonien kernel: Killed process 16503 (forkit) total-vm:1487720kB, anon-rss:342536kB, file-rss:3128kB, shmem-rss:0kB
Jun 21 06:07:05 balkonien kernel: Out of memory: Kill process 27684 (perl) score 10 or sacrifice child
Jun 21 06:07:05 balkonien kernel: Killed process 27702 (bzip2) total-vm:13624kB, anon-rss:6392kB, file-rss:952kB, shmem-rss:0kB
Jun 21 06:08:45 balkonien systemd[1]: Failed to start systemd-cron daily script service.

Output of ps faux (regarding the backup task):

samuel    9458  0.0  0.0  26016  2212 ?        Ss   Jun16   0:00 SCREEN -R
samuel    9459  0.0  0.0  25516  5812 pts/2    Ss   Jun16   0:00  \_ /bin/bash
root     14092  0.0  0.0  45336  3804 pts/2    S+   12:08   0:00      \_ sudo ./storeBackup
root     14093  0.0  0.0   4312   672 pts/2    S+   12:08   0:00          \_ /bin/sh ./storeBackup
root     14099 38.4  1.9 369208 306664 pts/2   SN+  12:08  33:13              \_ perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
root     14103  0.0  0.0  34752  7432 pts/2    SN+  12:08   0:00                  \_ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-54a4c1c781f71013 --out /var/log/storeBackup/storeBackup.log --maxFilelen 1000000 --noOfOldFiles 5
root     14116  1.6  0.0  13624  7500 pts/2    SN+  12:08   1:22                  \_ bzip2
root     14117  0.0  0.0  34732  7328 pts/2    SN+  12:08   0:00                  \_ perl /opt/storeBackup/lib/stbuLog.pl --readFile /tmp/prLog-bd0ccc7b5ac7d895 --out /media/Backup/storeBackup/default/2018.06.21_12.08.27/.storeBackup.log --maxFilelen 0 --noOfOldFiles 1

Output of iotop (first lines):

Total DISK READ :       7.06 K/s | Total DISK WRITE :       0.00 B/s
Actual DISK READ:      84.75 K/s | Actual DISK WRITE:       0.00 B/s
  TID  PRIO  USER     DISK READ  DISK WRITE  SWAPIN     IO>    COMMAND                                                                                                                                                                                                         
14099 be/7 root        7.06 K/s    0.00 B/s  0.00 %  0.06 % perl /usr/local/bin/storeBackup.pl -f /etc/storeBackup/storebackup.conf
    1 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % init
    2 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kthreadd]
    3 be/4 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [ksoftirqd/0]
    5 be/0 root        0.00 B/s    0.00 B/s  0.00 %  0.00 % [kworker/0:0H]

Output of top (first lines):

top - 13:38:35 up 5 days,  6:38,  1 user,  load average: 1,00, 1,15, 1,22
Tasks: 592 total,   2 running, 589 sleeping,   0 stopped,   1 zombie
%Cpu(s):  0,5 us, 25,7 sy,  1,5 ni, 71,8 id,  0,2 wa,  0,0 hi,  0,2 si,  0,0 st
KiB Mem:  15908456 total, 15733624 used,   174832 free,  2555436 buffers
KiB Swap: 12695892 total,   263052 used, 12432840 free. 10882640 cached Mem

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND                                                                                                                                                                                                    
14099 root      38  18  369208 306664   4920 R  44,5  1,9  34:54.76 perl                                                                                                                                                                                                       
   47 root      20   0       0      0      0 S   3,3  0,0  67:54.66 kswapd0                                                                                                                                                                                                    
18400 root      20   0   26224   3432   2568 R   2,0  0,0   0:00.21 top                                                                                                                                                                                                        
    7 root      20   0       0      0      0 S   1,6  0,0  29:58.23 rcu_sched                               

Output of free -h:

             total       used       free     shared    buffers     cached
Mem:           15G        14G       214M        18M       2,4G        10G
-/+ buffers/cache:       2,2G        12G
Swap:          12G       256M        11G

closed as too broad by Ipor Sircer, schily, Kusalananda, Thomas Nyman, slm Jun 22 '18 at 0:37

Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer. Avoid asking multiple distinct questions at once. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • instead of long useless log, you could had the result of the command ps faux, iotop top free -h that would help way more than your log. Also obviously your backup process eat all the capacity of the system. – Kiwy Jun 21 '18 at 7:53
  • @Kiwy it's not useless as it tells exactly what causes the problem. OOM killer is not a random magic - every process in linux has its calculated score and OOM killer pick the best one to kill. Simplifying is most likely it'll kill the one which consumes biggest amount of memory. More at this topic – pawel7318 Jun 21 '18 at 8:03
  • @pawel7318 it still doesn't inform you precissely of what is the cause for your problem, another program than the one causing the issue can be terminated. I already had the case several time. To me it's just an indication that something went wrong at that moment. make a conclusion on those log, is IMO not relevant, i'll read the doc you post though. – Kiwy Jun 21 '18 at 8:09
  • @Kiwy I added the outputs accordingly. Right now the backup script is only running for ten minutes. I can update the outputs in an hour, if it's neccessary. Or is it clear that the backup script is the cause and I should contact them? – Sammy Jun 21 '18 at 10:19
  • is the problem is currently happening ? because you have 11GB of cached memory that could be freed by the system before shooting processes. – Kiwy Jun 21 '18 at 10:32