My situation is virtually the same as this post on serverfault. To summarize:

I have a really simple C++ app that uses a hash-map to count and merge a number of large files. I reserve memory with with std::unordered_map::reserve() before filling it and start iterating. At about 60Gb of memory used (the system has 378Gb) the process starts swapping and the rate of hash insertions degrades into nothingness.

A barebones Gist of the program is here.

Some notes:

  • swapoff -a completely fixes the issue
  • swappiness is set to 1
  • Better hashmap implementations (like martinus/robin-hood-hashing) do not solve the issue.
  • Everything is compiled as 64bit, the only options to gcc are -std=c++11 -fopenmp -O3
  • The program is using 1 thread at this step
  • High I/O load on the server seems to worsen the issue

More info

file output for the executable:

../script/jelly_union: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=65917c99ec9480c1dfb859f10920fce15a8fefc1, not stripped

uname -a:

Linux picea 4.15.0-66-generic #75-Ubuntu SMP Tue Oct 1 05:24:09 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

CPU info (last core):

processor       : 63
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 1
model name      : AMD Opteron(tm) Processor 6282 SE
stepping        : 2
microcode       : 0x600063e
cpu MHz         : 1156.138
cache size      : 2048 KB
physical id     : 3
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 143
initial apicid  : 111
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid amd_dcm aperfmperf pni pclmulqdq monitor ssse3 cx16 sse4_1 sse4_2 popcnt aes xsave avx lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 nodeid_msr topoext perfctr_core perfctr_nb cpb hw_pstate ssbd ibpb vmmcall arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 5187.19
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb
  • You say that the purpose of your app is "to count and merge a number of large files". How much and what file I/O does your app do in this scenario? Mostly reading? Maybe man 2 posix_fadvise can help to reduce unnecessary caching. – johnLate Nov 1 '19 at 6:36
  • Thanks for the tip with fadvise, I didn't know about the possibility to drop caches with fadvise(). I tried to drop caches (echo 3 > /proc/sys/vm/drop_caches) during the run, which made no difference to the swapping issue, so I'm not sure it will help, but worth a try. A barebones version of the source is in this gist. Essentially it deserializes a binary file at once and stores it in two std::vector, then processes those to insert elements that pass a cutoff into the hash-map. – Bastian Schiffthaler Nov 1 '19 at 7:56
  • Why you need swap on machine with such amount of memory? It is totally useless. – gena2x Nov 1 '19 at 12:57
  • @gena2x In principle I agree, however there are certain workloads/apps that still like to have some swap. For the RAM I have it configured to 8G of swap. Some discussion on that topic is here or in the RedHat docs here. Also, while I admin these machines, we sometimes run on HPC clusters and I doubt they want to turn off their swaps for me... – Bastian Schiffthaler Nov 1 '19 at 13:53

I'm not certain what is the modern C++ way of doing it, but you might want to look into mlock(2) to disable swapping for memory pages.

I'm also somewhat curious as to what happens if you rerun it immediately. I would expect the reserve call you use do not actually touch all memory pages (and thus do not force them to be actually allocated) - if that was the case, what you start seeing after a while would be the system struggling to actually provide the memory you've asked for in the first place, not swapping it (you should be able to disprove this by looking at the resident size).

| improve this answer | |
  • Thanks for the input! I am experimenting with mlock() and it is correct that the libstd++ allocator does not actually touch the memory when reserve() is called. It's in VMEM but not residential. jemalloc does the same but tcmalloc actually touches the map. I'm closing this for now but thanks to everyone for their insight. – Bastian Schiffthaler Nov 5 '19 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.