1

I have noticed that some of my processes take a lot of virtual size in my computer. I understand that the virtual size of a process is the sum of memory it is actually using, memory it has mapped into itself (for instance the video card’s RAM for the X server), files on disk that have been mapped into it (most notably shared libraries), and memory shared with other processes. In other words, the virtual size represents how much memory the program is able to access at the present moment.

The processes I am running take a lot of virtual size (~117 GB), but little RAM:

enter image description here

and little GPU memory:

enter image description here

and low disk I/O:

enter image description here

Is there any potentially bad effect that such processes with high virtual size (but little RAM, little GPU memory, and low disk I/O) may have? Can they somehow slow down other processes? The computer has 32 GB of RAM.

4

In general, it probably won't have a bad effect. However, Linux by default allows overcommitting memory. That means that if a process asks for memory, Linux will say "sure". Then, if it actually runs out of memory (including swap space), Linux will start killing processes to free up memory.

So, if your process allocates 117GB but does not use most of it, it will show 117GB virtual memory. However, if your process then all of a sudden decides it's actually going to fill up that 117GB, Linux will run out of memory and kill it, and perhaps other processes as well (this is where the bad effect comes in).

The Linux kernel can be configured as to how much overcommitment it will allow. The process of how to do this is described in that link, but I'll describe it here as well to keep everything in one place.

On Linux kernels ≥ 2.5.30, there are two proc files that regulate this. First, there is /proc/sys/vm/overcommit_memory. This can have three values:

  • 0: let the kernel decide for itself how much overcommitment it allows
  • 1: allow unlimited overcommitment
  • 2: allow overcommitment according to /proc/sys/vm/overcommitment_ratio.

The other proc file, /proc/sys/overcommitment_ratio, denotes the percentage of overcommitment of memory it will allow if the other one is set to 2. If overcommit_memory is set to 2, Linux will allow all the swap space to be committed, plus overcommitment_ratio% of RAM.

  • Your link seems somewhat antiquated given it talks about 64-bit processors in a future sense. I assume the information is still valid even in current times where 64-bit processors are commonplace, however. – JAB Jun 13 '16 at 18:50
  • I would assume that memory overcommit is required for this workload. Some programs are written to allocate lots of memory but never use it in practice. Unless you're going to fix those programs you'll need to keep using the overcommit feature. In practice the difference between a program that allocates the memory but never uses it and a program that allocates exactly the amount it needs is that the latter may be a little slower. However, if former misbehaves and starts using all the allocated memory you'll end up with Out of Memory situation and kernel OOM Killer hopefully kills it. – Mikko Rantalainen Dec 17 '17 at 12:13
  • (continued) if a program that has allocated lots of memory only uses it when it really needs it will in fact work better than a program that allocates bits of memory as it needs because the latter program will need to do more syscalls which take a bit more CPU. With 64 bit CPUs the virtual memory is practically free so using it is not bad per se. – Mikko Rantalainen Dec 17 '17 at 12:16
  • Another term for this is an "optimistic malloc", in other words, programmatically, you can allocate memory for use, but the memory isn't used until it's actually populated. – Owl Nov 26 '18 at 13:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.