Top showing virtual memory usage of hundreds of gigabytes

Question

On a new virtual machine, top shows that a process of mine consumes 100 GB of virtual memory. I've read that it's possible because of overcommit, however, it's a bit too much. I've never seen anything like this, especially in a VM having just 6 GB real memory, 2 GB swap and a single 40 GB HD partition.

The funny process with 101 GB VIRT is Eclipse IDE (started a few minutes ago) and there's another one, WebKitWebProcess with 98 GB VIRT. All other processes have sane VIRT values of about 1-2 GB. All three tools seem to agree on that: top, htop and ps.

• Is there any good reason for the processes to request that much virtual memory?
• Isn't such behavior detrimental to the system?
• What are the current limits on the total virtual memory?

Update

Details:
Linux 4.15.0-38-generic x86_64 GNU/Linux, Linux Mint 19 Tara
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)
Version: 2018-09 (4.9.0) Build id: 20180917-1800

The content of /proc//maps starts with

00400000-00401000 r-xp 00000000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00600000-00601000 r--p 00000000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
00601000-00602000 rw-p 00001000 08:01 132092                             /usr/lib/jvm/java-8-oracle/jre/bin/java
014fc000-02ebe000 rw-p 00000000 00:00 0                                  [heap]
c0000000-d0000000 rw-p 00000000 00:00 0 
d0000000-100000000 ---p 00000000 00:00 0 
100000000-1012c6000 rw-p 00000000 00:00 0 
1012c6000-140000000 ---p 00000000 00:00 0 
7ef800000000-7ef800004000 rw-p 00000000 00:00 0 
7ef800004000-7ef8000ec000 rw-p 00000000 00:00 0 
7ef8000ec000-7ef800100000 rw-p 00000000 00:00 0 
7ef800100000-7f0800000000 rw-p 00000000 00:00 0 
7f0800000000-7f1000000000 ---p 00000000 00:00 0 

The last two shown lines line seem to be the culprit as 0x7f0800000000 - 0x7ef800100000 ≈ 68.7e9. From what I've found, it's an unmapped private region with no permissions. That's all I can say....

Answer 1

Most likely it is mmap'ing every file it can index. I've seen similar with Kafka mapping thousands of files but usually the virtual footprint is much higher (~10x) times). You can run:

cat /proc/{PID}/maps

(where {PID} is your Eclipse process ID) to see if this is the case. If it is indeed the case, you might have to raise vm.max_map_count in sysctl if your virtual footprint is too large.

Answer 2

Whenever mapping a file, the OS allocates a set of continuous virtual pages.

I don't specifically know about Java, but it is likely to map each new file at a new address which is aligned at least at a 1Mb boundary (most certainly 2Mb).

So if a very small file, say 64Kb, is mapped in memory, it will use, in continuous virtual pages, 2Mb of memory. Do that with enough small files and you quickly grow much faster than what the primary input is and as a result that can be larger than your disk and RAM.

---

Further, if running in some kind of debug mode, most memory allocation would be much larger. Although probably not your case here. For C/C++ we have a system developed by Google called the sanitizer which does that. It allows you to detect memory overflow (i.e. writing before the start of your block or after) and it does so using the MMU (i.e. so that you get a hardware interrupt when writing to memory at the wrong location; reading is not as well protected) which requires buffers to be at least at a 4K boundary on Intel processors. Those processes often appear to use a very large amount of memory such as 20Tb on startup.