3

I'm attempting to run an experiment for my masters thesis that involves an R script that has to run for quite a while (up to 24 hours of 100% cpu usage). It's a single-core process, and as far as I can tell I have ample memory for the job. The problem is that when I run it overnight, it's getting killed sometime in the middle of the night without my initiating anything. To give you an idea of the output, here's what I'm looking at:

➜  r-lda git:(master) ✗ Rscript slda-test-gibbs-1E6-01.r | tee slda-test-gibbs-1E6-01.log
Loading required package: lattice
Loading required package: ggplot2
Loading required package: methods
[1] "LOADING DATASET: data/split_1E6.dat"
[1] "DATASET LOADED"
[1] "TRAINING DATASET SIZE: 800000"
[1] "TESTING DATASET SIZE: 200000"
[1] "VOCAB SIZE: 129276"
[1] "TRAINING E=50 M=2 K=600"
[1]    17722 killed     Rscript slda-test-gibbs-1E6-01.r | 
       17723 done       tee slda-test-gibbs-1E6-01.log

I know that's not very informative, but I don't really know how to diagnose this symptom further. It's happened twice now, and I have no idea what's gonig on. I usually don't work much with R scripts, and a bit of googling didn't turn up any real explanation for this. To give you an idea of my system:

➜  r-lda git:(master) ✗ uname -a
Linux ****** 4.2.5-1-ARCH #1 SMP PREEMPT Tue Oct 27 08:13:28 CET 2015 x86_64 GNU/Linux


➜  r-lda git:(master) ✗ head -n26 /proc/cpuinfo
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 60
model name      : Intel(R) Core(TM) i7-4770K CPU @ 3.50GHz
stepping        : 3
microcode       : 0x12
cpu MHz         : 3499.863
cache size      : 8192 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtopology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl vmx est tm2 ssse3 fma cx16 xtpr pdcm pcid sse4_1 sse4_2 movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm ida arat epb pln pts dtherm tpr_shadow vnmi flexpriority ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt
bugs            :
bogomips        : 7015.72
clflush size    : 64
cache_alignment : 64
address sizes   : 39 bits physical, 48 bits virtual
power management:

➜  r-lda git:(master) ✗ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        3.4G         10G        291M        1.4G         11G
Swap:           15G        773M         15G

Any insight would be greatly appreciated. Thanks in advance.

  • 1
    Get the status code from the script, that'll tell you how the script is being killed, which is the first step towards knowing by who and why. The status code is in $?. You need the status code for the script, not for the whole pipeline. Knowing the exact time might also be a clue. You can use { Rscript slda-test-gibbs-1E6-01.r; echo "$? $(date)" >r.status; } | tee slda-test-gibbs-1E6-01.log – Gilles Nov 27 '15 at 23:27
3

The best starting point are logs. Since you say that the application log didnt show anything helpfull, you should exame system logs like dmesg and cron jobs. A common entry in dmesg would be an OOM (out of memory)-error but you said there was plenty of free memory. To get the time of a crash, my preferred way is to run

date; time command

where command is the programm to run. By adding the time output by time after the programm crashed to the time output by date when starting you get the time it crashed. Run this multiple times and look for patterns like same running time or crashing time. Also look at daily cronjobs which are not started at the same time every day.

Other reasons might be an bug in the program or the programm using 32 bit pointer and failing to allocate more memory.

  • You're totally right. I just grepped dmesg and saw that it was an OOM error. Apparently the algorithm doesn't preallocate the way I thought it did. Thanks. – Axel Magnuson Nov 28 '15 at 0:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.