Weblogic process just killed suddenly on redhat linux

Question

One of the process on my WAS killed at dawn.

6 process run on this server. But only one process just killed.

I don't know how to fix it. This is Weblogic last log that before process killed.

/app/orarecp/bea/domains/HRRECP/bin/startWeblogic.sh: line 180: 27066 killed               ${JAVA_HOME}/bin/java ${JAVA_VM} {MEM_ARGS} -Dweblogic.Name=${SERVER_NAME} -Djava-security.policy=${WL_HOME}/server/lib/weblogic.policy ${JAVA_OPTION} ${PROXY_SETTINGS} ${SERVER_CLASS}

And This is startWeblogic shell script. startWeblogic Shell Script: 180 line

And This is start process shell script parts.

USER_MEM_ARGS="-Xms3072m -Xmx3072m -Xverbose:gc -Xverboselog:${LOG_HOME}/gclogs/gc.${SERVER_NAME}_`date + '%Y%m%d_%H%M%S'`.out"

JAVA_OPTION="${JAVA_OPTIONS} -Xbootclasspath/p:/app/orarecp/jennifer/agent/lwst.jdk.jar:/app/orarecp/jennifer/agent/lwst.boot.jar:/app/orarecp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/orarecp/jennifer/agent/R11.conf -javaagent:/app/orarecp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom"

server specification is...

• CPU: 8 core, 2.4GHz // The time that CPUs utilization was under 15%
• Memory: 32Gb
  • Mem: 32922024k total, 32817492k used, 104580k free, 147464k buffers
  • Swap: 33551744k total, 148k used, 33551596k free, 15231864k cached

server stack is...

• weblogic 11g
• jrockit 1.6
• redhat linux 5.8 (64bit)

I wonder why the process is killed. What am I suppose to do?

Please answer if you need additional information.

All the answers are going to be helpful to me.

Thanks.

---

Additinal Information.

1. before that Last log But this log is occurs frequently.

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception

java.lang.NullPointerException at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

<[ServletContext@228988411[app:Recruit-Front module:WebContent path:/recruit spec-version:2.5]] Servlet failed with Exception java.lang.NullPointerException

at cj.recruit.syscommon.filter.SessionLocaleFilter.doFilter(SessionLocaleFilter.java:41) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at frameone.core.common.web.filter.EnvironmentSetupFilter.doFilter(EnvironmentSetupFilter.java:88) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at cj.recruit.syscommon.filter.CrossScriptingFilter.doFilter(CrossScriptingFilter.java:22) at weblogic.servlet.internal.FilterChainImpl.doFilter(FilterChainImpl.java:56) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.wrapRun(WebAppServletContext.java:3730) at weblogic.servlet.internal.WebAppServletContext$ServletInvocationAction.run(WebAppServletContext.java:3696) at weblogic.security.acl.internal.AuthenticatedSubject.doAs(AuthenticatedSubject.java:321) at weblogic.security.service.SecurityManager.runAs(SecurityManager.java:120) at weblogic.servlet.internal.WebAppServletContext.securedExecute(WebAppServletContext.java:2273) at weblogic.servlet.internal.WebAppServletContext.execute(WebAppServletContext.java:2179) at weblogic.servlet.internal.ServletRequestImpl.run(ServletRequestImpl.java:1490) at weblogic.work.ExecuteThread.execute(ExecuteThread.java:256) at weblogic.work.ExecuteThread.run(ExecuteThread.java:221)

2. another JAVA process (ps -ef | grep java)

ps -ef | grep java

3. another Server Java Process

   ps -ef | grep java

oraehrp 818 786 0 19:34 pts/1 00:00:00 grep java

root 9689 1 1 2015 ? 1-01:05:58 /usr/java/jdk1.6.0_23/bin/java -Djava.awt.headless=true -jar /app/CJ_agent//cjagent.jar /app/CJ_agent//conf/cjagent.conf

oraehrp 10128 10074 2 2015 ? 1-10:06:08 /usr/java/jrockit1.6/bin/java -jrockit -Xms1024m -Xmx1024m -Dweblogic.Name=EHRREP1 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 14795 14740 2 2015 ? 1-10:31:48 /usr/java/jrockit1.6/bin/java -jrockit -DAdminServer -Xms1024m -Xmx1024m -verbosegc -Dweblogic.Name=AdminServer -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=true -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 23456 23395 67 18:25 ? 00:46:23 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD11/gclogs/gc.EHRPRD11_20160201_182544.out -Dweblogic.Name=EHRPRD11 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E11.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 24233 24179 49 18:27 ? 00:33:03 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD12/gclogs/gc.EHRPRD12_20160201_182747.out -Dweblogic.Name=EHRPRD12 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehrp/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E12.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

oraehrp 25006 24944 53 18:29 ? 00:34:35 /usr/java/jrockit1.6/bin/java -jrockit -Xms2048m -Xmx2048m -Xns:256m -Xverbose:gc -Xverboselog:/svcroot/logs/was/domains/HREHRP/EHRPRD13/gclogs/gc.EHRPRD13_20160201_182933.out -Dweblogic.Name=EHRPRD13 -Djava.security.policy=/app/oraehrp/bea/wlserver_10.3/server/lib/weblogic.policy -Dweblogic.ProductionModeEnabled=true -Dweblogic.security.SSL.trustedCAKeyStore=/app/oraehrp/bea/wlserver_10.3/server/lib/cacerts -Dweblogic.threadpool.MinPoolSize=100 -Xbootclasspath/p:/app/oraehRP/jennifer/agent/lwst.jdk.jar:/app/oraehrp/jennifer/agent/lwst.boot.jar:/app/oraehrp/jennifer/agent/jennifer.boot.jar -Djennifer.config=/app/oraehrp/jennifer/agent/E13.conf -javaagent:/app/oraehrp/jennifer/agent/lwst.javaagent.jar -Dframeone.webapps=/svcroot/runtime/webapps -Dfile.encoding=UTF-8 -Djava.security.egd=file:///dev/urandom -da -Dplatform.home=/app/oraehrp/bea/wlserver_10.3 -Dwls.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.home=/app/oraehrp/bea/wlserver_10.3/server -Dweblogic.management.discover=false -Dweblogic.management.server=t3://52.2.205.138:7001 -Dwlw.iterativeDev=false -Dwlw.testConsole=false -Dwlw.logErrorsToConsole=false -Dweblogic.ext.dirs=/app/oraehrp/bea/patch_wls1036/profiles/default/sysext_manifest_classpath:/app/oraehrp/bea/patch_ocp371/profiles/default/sysext_manifest_classpath weblogic.Server

Answer 1

This is due to lack of memory. You have slightly less than 300MB RAM free (free+buffers), and each new weblogic Java instance will be eating 3GB.

There is a functionality of the Linux kernel called the OOM killer that kills a random process whenever there is insufficient memory.

How to Configure the Linux Out-of-Memory Killer

Have a look at that JAVA arguments, and at runaway Java Scripts. You are using a lot of RAM.

Evaluate whether you need to fine tune the Java Xms/Xmx or add more RAM to the server.

I have had the opportunity to check WebLogic requirements now. As Customizing the WebLogic JVM heap size states:

The default JVM heap size for WebLogic is 3GB. ... USER_MEM_ARGS="-Xms128m -Xmx3072m ...

Pay attention to the 128m, the recommended configuration is giving it 128m when starting the process, and it growing as needed to 3GB.

Xms is the minimum memory allocated, Xmx the maximum memory. If you change Xms to 128m, the threads will only use up the memory they need when starting, and not 3GB at once by default. That will probably use less RAM over time.

From What are the Xms and Xmx parameters when starting JVMs?

The flag Xmx specifies the maximum memory allocation pool for a Java Virtual Machine (JVM), while Xms specifies the initial memory allocation pool. The memory flag can also be specified in multiple sizes, such as kilobytes, megabytes, and so on.

I would also evaluate after this change whether you need more RAM, as that the value you have now is not enough for a proper operation (e.g. too low).