7

I am currently having some issues running Java. It won't start because of heap issues. But I have more than 9 GB Ram free (or even 16 GB if you assumed the cache would be empty). This is the error I get (and the free command)

root@server: ~ # java
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

root@server: ~ # free
             total       used       free     shared    buffers     cached
Mem:      25165824   15941148    9224676          0          0    7082176
-/+ buffers/cache:    8858972   16306852
Swap:            0          0          0

I am running a 64 bit Debian on a virtualized server. The virtualization software is OpenVZ. This is my Java version (I can execute this command after I stop two of my VM's (4 currently running)):

root@server: ~ # java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)

What could I do?


As requested:

root@server: ~ # cat /proc/meminfo
MemTotal:       25165824 kB
MemFree:        11723412 kB
Cached:          4597552 kB
Active:          9692308 kB
Inactive:        3322544 kB
Active(anon):    7411960 kB
Inactive(anon):  1005340 kB
Active(file):    2280348 kB
Inactive(file):  2317204 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:               960 kB
Writeback:             0 kB
AnonPages:       8417300 kB
Shmem:             21504 kB
Slab:             427452 kB
SReclaimable:     383424 kB
SUnreclaim:        44028 kB

Reguest2:

root@server: ~ # cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 10023468:  kmemsize                399250512            506245120           5053325720           5558658292                    0
            lockedpages                     0                    8               246744               246744                    0
            privvmpages               6005602              6291447              6291456              6291456                  221
            shmpages                     8576                 8608               579124         579124289562                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       598                 1236                30000                30000                    0
            physpages                 4634494              6291456              6291456              6291456                    0
            vmguarpages                     0                    0              6291456  9223372036854775807                    0
            oomguarpages              1529371              2144671              6291456  9223372036854775807                    0
            numtcpsock                     62                  164                30000                30000                    0
            numflock                       25                   39                 1000                 1100                    0
            numpty                         13                   24                  512                  512                    0
            numsiginfo                     10                   75                 1024                 1024                    0
            tcpsndbuf                 3330352              4153232           1179110194           1684441906                    0
            tcprcvbuf                 1216896             34410032           1179110194           1684441906                    0
            othersockbuf               270504               537552            589555096           1094886808                    0
            dgramrcvbuf                     0                67048            589555096            589555096                    0
            numothersock                  287                  333                30000                30000                    0
            dcachesize              355559855            446054103           1103879952           1136996352                    0
            numfile                      4766                 7745               250000               250000                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      14                   14                 1000                 1000                    0

root@server: ~ # java
Error occurred during initialization of VM
Could not reserve enough space for object heap
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

root@server: ~ # cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
 10023468:  kmemsize                399246622            506245120           5053325720           5558658292                    0
            lockedpages                     0                    8               246744               246744                    0
            privvmpages               6005601              6291447              6291456              6291456                  233
            shmpages                     8576                 8608               579124         579124289562                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numproc                       598                 1236                30000                30000                    0
            physpages                 4635460              6291456              6291456              6291456                    0
            vmguarpages                     0                    0              6291456  9223372036854775807                    0
            oomguarpages              1529376              2144671              6291456  9223372036854775807                    0
            numtcpsock                     64                  164                30000                30000                    0
            numflock                       25                   39                 1000                 1100                    0
            numpty                         13                   24                  512                  512                    0
            numsiginfo                     10                   75                 1024                 1024                    0
            tcpsndbuf                 3365232              4153232           1179110194           1684441906                    0
            tcprcvbuf                 1249664             34410032           1179110194           1684441906                    0
            othersockbuf               270504               537552            589555096           1094886808                    0
            dgramrcvbuf                     0                67048            589555096            589555096                    0
            numothersock                  287                  333                30000                30000                    0
            dcachesize              355559855            446054103           1103879952           1136996352                    0
            numfile                      4768                 7745               250000               250000                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            dummy                           0                    0  9223372036854775807  9223372036854775807                    0
            numiptent                      14                   14                 1000                 1000                    0
  • Can you cat /proc/meminfo – Matt Jan 16 '14 at 19:23
  • @mindthemonkey Post edited. It seems to me that there is enough space for the default size. – BrainStone Jan 16 '14 at 20:05
  • Certainly looks that way =) You are trying to run the java process in the container, right? I assume you can run the java process in the host successfully? How much memory is allocated to each container vzmemcheck -vA? Can you cat /proc/user-beancounter before and after the error. – Matt Jan 16 '14 at 21:10
  • @mindthemonkey I do not have acces to anything outside the container... – BrainStone Jan 16 '14 at 22:54
  • ok.. cat /proc/user-beancounter is done in your container. – Matt Jan 16 '14 at 23:41
9

OpenVZ & Memory

The failcnt is going up on privvmpages, so your container is unable to allocate any more virtual memory space from the host:

root@server: ~ # cat /proc/user_beancounters
Version: 2.5
       uid  resource                     held              maxheld              barrier                limit              failcnt
            privvmpages               6005601              6291447              6291456              6291456                 >233<
            physpages                 4635460              6291456              6291456              6291456                    0
            vmguarpages                     0                    0              6291456  9223372036854775807                    0
            oomguarpages              1529376              2144671              6291456  9223372036854775807                    0

Note that virtual memory != physical memory. Processes can allocate up to somewhere around the addressable amount of virtual memory (32bit ~ 2G - 4G, 64bit 8 TB - 256 TB) but that doesn't mean physical memory pages are being used ( a page being a 4KB chunk of memory).

physpages is the number of physical memory pages your container can use.
oomguarpages is the guaranteed memory pages the container will receive when the host is memory constrained.
privvmpages is the number of virtual memory pages your container can use
vmguarpages is the guaranteed amount of virtual memory in the same way

Java

Oracle Java will always allocate one contiguous chunk of virtual memory. Running java with no arguments on a box results in 5M of real memory used (RSS), but 660M of VM space allocated (VSZ):

  PID COMMAND                        VSZ   RSS
20816 java                        667496  4912 

Looking at the memory segments for the java process in it's smaps file shows a chunk of about 500MB allocated, the rest is memory mapped files and normal java stuff.

On a system that's been up for a while the available VM space becomes fragmented as processes use/free parts of it. A grep Vmalloc /proc/meminfo will give you VmallocChunk which is the largest free chunk currently available. If this is low, the system will try and allocate more when java requests it, after all it's virtually unlimited on a 64bit box.

Fix

Tell your host to configure privvmpages and vmguarpages much higher. There's no need for them to be the same as physical memory as that impacts the way linux memory works

You might be able to work around the problem temporarily by dropping your file cache echo 1 > /proc/sys/vm/drop_caches but that's only temporary.

You can limit the chunk of memory java tries to allocate at run time with a minimum Xms or while running with maximum Xmx. Running java with these options on my machine:

java -Xms10M -Xmx10M

reduces the total virtual size to 140MB or so with only a 10MB contiguous chunk for the java heap allocated.

  • Thank you for your answer. I am 100% sure that my hoster is not going to change anything about the container. I also cannot drop my cache because my container is on OpenVZ. (In addition grep Vmalloc /proc/meminfo gives no output) Running java -Xms10M -Xmx10M works though. But this does not really solve all problems. Like starting additional programms with a RAM usage of something arround 2GB... – BrainStone Jan 19 '14 at 16:46
  • Why are you using them if you don't think their support wouldn't help you with a config issue? Maybe site the config examples and tables? It doesn't cost them anything to allocate extra unusable memory pages. – Matt Jan 19 '14 at 18:55
  • I use them because the quality is sufficient and the price is low. Second is there a way to tell Java it should not use continous chunks of memory? – BrainStone Jan 19 '14 at 21:38
  • @BrainStone You are running a 64 bit JVM. The fragmentation limitation should only affect 32 bit applications on a 32 bit OS. – jlliagre Jan 20 '14 at 22:11
  • @jlliagre openvz is putting an artificial limit on the VM space. – Matt Jan 21 '14 at 19:07
6

In our case it helped to constrain the heap size that the VM tries to reserve on startup.

For example on the command line:

export _JAVA_OPTIONS='-Xms64M -Xmx128m'

or for Tomcat in [TOMCAT_HOME]/bin/setenv.sh

#!/bin/sh

JAVA_OPTS="-Xms64M -Xmx256M"

(Note the extra leading underscore on the command line.)

The explanation of our provider is:

  • the java VM calculates an initial heap size on startup
  • this calculation is based upon the RAM available.
  • some java installations on virtual private servers (VPS) fail to reflect the memory constrains of the VPS when calculating the initial heap size. Instead they base their calculation upon the memory size of the hosting system. Which may lead to an amount of free memory that is not available on the VPS.
  • when providing an initial heap size via JAVA_OPTS the VM uses these values and does not try to calculate on its own.
  • This fixed my problem. I wasn't on a virtual server, rather on hardware with 80G of RAM. The server is new and not used yet, so all that memory was available. The default system memory limit of 16GB caused me to be unable to start the JVM. Even as root. Modifying /etc/security/limits.conf fixed the problem. RHEL6.6. – Dan Pritts Jan 16 '15 at 18:13
2

Probably you need to change the default java heap size, try out these arguments:

-Xms<size>        set initial Java heap size
-Xmx<size>        set maximum Java heap size
-Xss<size>        set java thread stack size

for example:

java -Xms64m -Xmx1512m

This thread could be helpful

Also check the ulimit (ulimit provides control over the resources available to the shell and to processes started by it):

ulimit -a 
  • I'm already using the heap size arguments (when starting applications). And my ulimit -a seems normal. Most values including virtual memory are unlimited. The ones that seemed a little low I doubled (-s from 8192 to 16384 and -l from 64 to 128) for testing but that didn't help either. – BrainStone Jan 17 '14 at 23:14
  • And what about the address space size for the process (-A) ? One more thing, it looks like you don't have any swap space at all. That's really not good as swap also adds to your virtual memory size. Java will try to allocate the memory in a big chunk, so the OS might try to re-allocate some unused memory to swap. Please try to add 32GB of swap. – b13n1u Jan 17 '14 at 23:50
  • There is no -A option for me. And I'm not allowed to add Swap. – BrainStone Jan 19 '14 at 16:40
2

You might just be out of virtual memory.

You are focused on RAM usage (or lack of) but your existing JVMs are reserving virtual memory, not RAM. As long as reserved pages of memory are not accessed, there is no RAM usage reported for them.

While Linux usually mask this issue by its over-commitment settings, it is quite possible that the fact you are running in an OS level virtualized environment either reduce/disable over commitment or set a virtual memory resource control capping.

On a regular OS instance (not a container), the easiest way to workaround this is to add some swap which will allow the reserved memory not to waste RAM. On a container, the way to overcome the issue might also require increasing the globally available virtual memory but more generally is simply done by granting more virtual memory to the container.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.