Skip to main content
Feb
Feb
Participating Frequently
May 22, 2014
Question

CF10 on CentOS 6.5 memory/CPU issues

  • May 22, 2014
  • 2 replies
  • 3350 views

I have been playing with CF10 performance on the CentOS 6.5 environment and so far I have lots of issues with it. I really appreciate any help on resolving the issues. It is upgraded version of CF from 8 to 10, both are 64-bit.

It seems to be memory leaks issue with CF so I have tried to adjust jvm.config as well as when ahead and upgraded CF's java version to the 1.7.0.55 version (tried this solution, thread (Re: SEVERE: The web application [/] created a ThreadLocal). Some of the issues were resolved but not all, the CF still eats memory then gets stuck. I have tried to use the following jvm settings:

Old one:

----------------------------------------------------------------------

java.args=-server  -Djava.awt.headless=true -Xms384m -Xmx1152m -Xmn128m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=384m -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -javaagent:/opt/coldfusion10/newrelic/newrelic.jar -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

----------------------------------------------------------------------

Here is the new one:

----------------------------------------------------------------------

java.args=-server  -Djava.awt.headless=true -Xms1280m -Xmx1280m -Xmn128m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=256m -XX:+UseConcMarkSweepGC -XX:NewSize=48m -XX:SurvivorRatio=4 -XX:+UseParNewGC -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dsun.rmi.dgc.client.gcInterval=600000 -Dsun.rmi.dgc.server.gcInterval=600000 -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars

----------------------------------------------------------------------

still no luck, at this time I am getting the following errors:

----------------------------------------------------------------------

Exception in thread "DefaultQuartzScheduler_QuartzSchedulerThread" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-41" java.lang.OutOfMemoryError: Java heap space

Exception in thread "Timer-2" Exception in thread "ajp-bio-8012-exec-26" java.lang.OutOfMemoryError: Java heap space

java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-43" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-42" java.lang.OutOfMemoryError: Java heap space

Exception in thread "ajp-bio-8012-exec-45" java.lang.OutOfMemoryError: Java heap space

May 22, 2014 3:14:46 AM org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [/] threw exception [ROOT CAUSE:

java.lang.OutOfMemoryError: Java heap space

] with root cause

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: Java heap space

        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70)

        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)

        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

        at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)

        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)

        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

----------------------------------------------------------------------

and the CF should be restarted.

CPU: 2.20GHz (2 cores)

RAM: 16Gb

The CF handles approximately 90 CF-based sites. Any help would be appreciated.

Thanks!

    This topic has been closed for replies.

    2 replies

    C
    carl type3
    Legend
    May 22, 2014

    Hi Feb,

    CF on CentOS is not my particular environment but it is Java so the environment difference should not matter.

    Some comments for you to consider:

    Max heap setting -Xmx1280m could you be more generous with 16Gb total RAM. Maybe -Xmx6144m


    Sometimes setting initial and maximum the same is good sometimes bad. One would need to perform some logging to know if the heap is starting to have contiguous memory space problems.


    On both examples you have used a mix of garbage collectors that would be fair to say tend to occupy memory more. Sometimes keeping object in memory is desirable however while you are having issues it might be suitable to change
    garbage collection to something that is fair to say more prone to evacuating objects from the heap. CF by default installs with -XX:+UseParallelGC which I often change myself tho given your issue you might be better using for now.


    You seem to have two parameters specifying “new” size of heap space in second example.

    So perhaps you could try this suggestion for JVM args. If that does not suffer from java.lang.OutOfMemoryError: Java heap space problems then enable some logging to see what other adjustments to memory or garbage collection would be more suitable. As always take a backup copy of JVM.CONFIG before applying any changes.

    java.args=-server -Djava.awt.headless=true -Xms2048m -Xmx6144m -Xmn128m -Dsun.io.useCanonCaches=false -XX:PermSize=256m -XX:MaxPermSize=512m -XX:+UseParallelGC -XX:ParallelGCThreads=2 -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application .home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/j ars,{application.home}/wwwroot/WEB-INF/cfform/jars

    HTH, Carl.

    Feb
    FebAuthor
    Participating Frequently
    May 23, 2014

    Hello Carl,

    My thought was the same as yours, but I have done as of yet because this value is really high. I will try it and let you know.

    I know these JVM settings do not look an accurate or production settings. I though, I resolved the issue, but it arose again so I have gone astray.

    My testing rules was as follow:

    java.args=-server -Xms2048m -Xmx2048m -XX:MaxPermSize=256m -XX:NewRatio=4 -XX:SurvivorRatio=8 -XX:+UseCompressedOops -Xss256k -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+DisableExplicitGC -XX:+UseCMSInitiatingOccupancyOnly -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:CMSInitiatingOccupancyFraction=68 -XX:TargetSurvivorRatio=90 -Xmn1024m -Dsun.io.useCanonCaches=false -XX:+CMSParallelRemarkEnabled -XX:ParallelGCThreads=2 -XX:PermSize=192m -Djava.awt.headless=true -Xbatch -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcoldfusion.session.protectfixation=false

    Ok, thank you for the suggestions, I will try it now and let you know.

    Best regards,

    Feb

    C
    carl type3
    Legend
    May 24, 2014

    Hello Feb,

    >My testing rules was as follow
    Sure testing can be one thing production another. I have used similar JVM settings to those before in production and found those are good for keeping used objects allocated, most of the heap is collected with minor garbage collections (GC), pause effect is not noticeable and while you may get a full GC, a java stop all process, full GC would not be frequent and short duration since objects not referred to are well taken care of by the minor GC.

    Having said that I don't have that type of GC in production  preferring to use garbage first (G1GC) since Java 1.7.0_40, G1GC being suitable to high object allocation,
    low pause effect  for full GC are not done often and are short in duration. Plus the  JVM args are easier to read with about 4 G1GC parameters versa say 12 for the “testing” combination.

    As for easy reading of JVM args Java 8 makes that even simpler for 8 does not use PermGen, tho CF does not list support for 8 yet so perhaps the less said about that the better.

    Lets see if the Java memory errors cease with the sanity UseParallel setting then perhaps with some Java logging work forward from there to increase the performance of JVM.

    Kind Regards, Carl.

    Anit_Kumar
    Community Manager
    Anit_KumarCommunity Manager
    Community Manager
    May 22, 2014

    Hi Feb,

    There are two concerns over here:

    From the error below, Sever Monitoring seems to be turned on and causing OutOfMemory error. Turn it off ( Server Monitoring > Settings, disable all). On a production box, we recommend to turn monitoring off, as it consumes lot of memory resources.

    Feb wrote:

    javax.servlet.ServletException: ROOT CAUSE:

    java.lang.OutOfMemoryError: Java heap space

            at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:70 )

            at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

            at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j ava:243)

            at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)

            at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)

            at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)

            at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)

            at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)

            at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)

            at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)

            at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)

            at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:414)

            at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:204)

            at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.jav a:539)

            at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:298)

            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

            at java.lang.Thread.run(Thread.java:745)

    ----------------------------------------------------------------------

    Secondly, I noticed that you are running 90 CF based websites. You may have to, tune your ColdFusion connectors as well. You may refer to ColdFusion 11 IIS Connector Tuning — Adobe ColdFusion Blog

    Feb wrote:

    The CF handles approximately 90 CF-based sites. Any help would be appreciated.

    Thanks!

    The jvm.config also needs tuning as well.

    Feb
    FebAuthor
    Participating Frequently
    May 23, 2014

    Hello Anit,

    Thank you for the reply. I really appreciate it.

    It is definitely disabled. I already encountered with those monitoring issues when there was enabled this setting "Enable Memory Tracking".

    I am sorry I forgot to mention regarding the CF connectors. Here are the settings:

    /opt/coldfusion10/config/wsconfig/1/workers.properties:

    worker.cfusion.max_reuse_connections=250

    worker.cfusion.connection_pool_size=1000

    worker.cfusion.connection_pool_timeout=60

    /opt/coldfusion10/cfusion/runtime/conf/server.xml:

    <Connector executor="tomcatThreadPool" maxThreads="1000" port="8500" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="60000" redirectPort="8445"/>

    Best regards,

    Feb

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices