Coldfusion 9 random slowness

[Sasha365i]

Hi guys and gals,

I have a Coldfusion 9 installation with MySQL 5.5 on Windows Server 2008 R2 that is about 2 years old. Mostly problem free. Reasonably well configured with very fast page loading times.

Recently it began experiencing intermittent slowdowns. It would run fast for days (both peak days and weekends) with practically no CPU consumption (in low teens usually), then all of a sudden it would spike up to 80-100% and would run like that for days (both peak and weekends). Then after being slow for days it would return back to normal on its own.

I've been trying to isolate the problem with MySQL, but all queries are optimized and debug output shows then as executing in 0 seconds or being cached. Yet page loading times are high.

The logs in C:\ColdFusion9\runtime\logs show nothing unusual. Memory tracking in server monitor is turned OFF.

Any suggestions how to hunt down and troubleshoot this random slowness would be greatly appreciated!

Many thanks

Sasha.

[carl type3]

Sasha, It is JRun.EXE that is utilising the 80-90% of CPU?

You say CF Server Monitor is present so I expect Enterprise licence. Without Memory Tracking enabled, Server Monitor > Statistics > Memory Usage > Memory Usage Summary is still available and while it is not showing all the CF JVM memory spaces or provide much detail on garbage collection it may provide a hint if there is some kind of New or Old memory issue. Perhaps you can attach a picture how Memory Usage Summary looks when system is operating normal versa when CPU is bound.

You could try JVM logging to get some more details and see if there is a garbage collection or memory problem with one of the New, Old or Permanent memory spaces. Let me know if you want to take the time to enable that and need some know how then I will post details.

HTH, Carl.

[Sasha365i]

Carl, thanks for your reply.

I can enable memory tracking. What should I be looking for once enabled? Also how do I do JVM logging?

Thank you!

[carl type3]

Not sure CF Mon Memory Tracking is going to assist so I will leave that to other responders to thread case there is something to gain.

Some questions CF9 update level and hotfix? 64 or 32 bit? CF Java version eg 1.6.0_17?

So on to JVM logging. Backup or take copy of JVM.CONFIG file. Edit JVM CONFIG and add this syntax without CR/LF to Arguments  section (copy pasting to from forum page can be odd take care):

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

Restart CF application service for JVM.CONFIG to be re-read and logging to commence.

Creates a log file in ColdFusion9\runtime\bin\cfjvmGC.log Jrun4\bin\ in case multiserver. The log details can be a bit hard to understand so use a tool eg GCViewer to graphically examine the log file contents:

http://www.tagtraum.com/gcviewer.html

I suspect GCViewer display of log file from when system is working normally to when CPU is bound will look significantly different so post back your findings. If not then likely not a CF JVM issue, something you can discount as issue, move on for other causes and at worst something learned. Remember to switch logging off when you are finished with it by removing syntax from JVM.CONFIG and restarting CF.

JVM.CONFIG syntax example only with JVM logging turned on:

# Arguments to VM

java.args=-server -Xmx512m -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log -Dcoldfusion.rootDir=etc

HTH again, Carl.

[Sasha365i]

Carl,

Thanks for your reply. The problem is intermittent in nature and I have no server admin skills or courage to play around with CF installation on a level you suggest.

Here is a screenshot of what a prolonged CPU and page load spike looks like. Today, Monday is a high traffic day and all day the site was super fast with pages loading around 40ms. Then *something* happens. Unrelated to the traffic. And CPU spikes and page load times go up.

I have enabled memory tracking and jvm logging temporarily but could not see anything unusual there. Also nothing unusual in CF/runtime/logs

I should add this machine is on a virtual cloud with 4 CPUs and 8GB of RAM. Any suggestions would be greatly appreciated.

[carl type3]

Fair enough you don't want to muck about with the internals of CF / Java however seems from the CF Monitor Overview the Reports JVM (used /max) 472 is close to 512M which could be leading to a lot of garbage collections causing Jrun to accumulate CPU time and GC pausing (full GC being a Java stop all process) applications.

From CF Monitor the Memory Usage Summary diagram would be interesting to look at when the response times drop?

Is Windows and CF 32 or 64 bit?

Without evidence (logs) it can be hard to say do this or that. Guessing makes me think you might just need to define a bit more memory to heap. So that is :

Server Settings > Java and JVM

Maximum JVM Heap Size (MB)

1024 in place of 512

Well that is a long way from CF Java tuning and there is the Server Settings > Java and JVM > JVM Arguments > XX:MaxPermSize= to consider but if the larger heap gets you out of trouble then fine.

HTH, Carl.

[Sasha365i]

Hi Carl,

Thanks for your reply.

The JVM size is actually set at 1024.

JVM Arguments are:

-server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

Which logs do you suggest I look at?

Thanks,

Sasha.

[carl type3]

JVM max is 1024 odd when the CF MON Summary says 472 of 512, I would expect the max to increase. A picture of CF MON memory summary would be good to know. What is Server Settings > Java and JVM > Minimum JVM Heap Size (MB) value?

Maybe you can try Server Settings > Java and JVM > JVM Arguments > -XX:PermSize=192m -XX:MaxPermSize=256m in place of -XX:MaxPermSize=192m

Sasha asks logs – that’s me referring to JVM logging to get a look at what is happening. I would rather make an informed suggestion at what to change rather than having a guess at it.

What does CFadmin > System Information (i button) say about Java Version and Java VM Name?

Regards, Carl.

[Sasha365i]

System Information
Server Details
Server Product	ColdFusion
Version	9,0,0,251028
Edition	Enterprise
Serial Number	[removed]
Operating System	Windows Server 2008
OS Version	6.1
Adobe Driver Version	4.0 (Build 0005)
JVM Details
Java Version	1.6.0_14
Java Vendor	Sun Microsystems Inc.
Java Vendor URL	http://java.sun.com/
Java Home	C:\ColdFusion9\runtime\jre
Java File Encoding	Cp1252
Java Default Locale	en_US
File Separator	\
Path Separator	;
Line Separator	Chr(13)
User Name
User Home	C:\
User Dir	C:\ColdFusion9\runtime\bin
Java VM Specification Version	1.0
Java VM Specification Vendor	Sun Microsystems Inc.
Java VM Specification Name	Java Virtual Machine Specification
Java VM Version	14.0-b16
Java VM Vendor	Sun Microsystems Inc.
Java VM Name	Java HotSpot(TM) 64-Bit Server VM
Java Specification Version	1.6
Java Specification Vendor	Sun Microsystems Inc.
Java Specification Name	Java Platform API Specification
Java Class Version	50.0
CF Server Java Class Path	;C:/ColdFusion9/runtime/../lib/activation.jar;  C:/ColdFusion9/runtime/../lib/ant-launcher.jar;  C:/ColdFusion9/runtime/../lib/ant.jar;  C:/ColdFusion9/runtime/../lib/antlr-2.7.6.jar;  C:/ColdFusion9/runtime/../lib/apache-solr-core.jar;  C:/ColdFusion9/runtime/../lib/apache-solr-solrj.jar;  C:/ColdFusion9/runtime/../lib/asn1.jar;  C:/ColdFusion9/runtime/../lib/axis.jar;  C:/ColdFusion9/runtime/../lib/backport-util-concurrent.jar;  C:/ColdFusion9/runtime/../lib/bcel.jar;  C:/ColdFusion9/runtime/../lib/bcmail-jdk14-139.jar;  C:/ColdFusion9/runtime/../lib/bcprov-jdk14-139.jar;  C:/ColdFusion9/runtime/../lib/cdo.jar;  C:/ColdFusion9/runtime/../lib/cdohost.jar;  C:/ColdFusion9/runtime/../lib/certj.jar;  C:/ColdFusion9/runtime/../lib/cf-acrobat.jar;  C:/ColdFusion9/runtime/../lib/cf-assembler.jar;  C:/ColdFusion9/runtime/../lib/cf-logging.jar;  C:/ColdFusion9/runtime/../lib/cf4was.jar;  C:/ColdFusion9/runtime/../lib/cf4was_ae.jar;  C:/ColdFusion9/runtime/../lib/cfusion-req.jar;  C:/ColdFusion9/runtime/../lib/cfusion.jar;  C:/ColdFusion9/runtime/../lib/clibwrapper_jiio.jar;  C:/ColdFusion9/runtime/../lib/commons-beanutils-1.8.0.jar;  C:/ColdFusion9/runtime/../lib/commons-codec-1.3.jar;  C:/ColdFusion9/runtime/../lib/commons-collections-3.2.1.jar;  C:/ColdFusion9/runtime/../lib/commons-digester-2.0.jar;  C:/ColdFusion9/runtime/../lib/commons-discovery-0.4.jar;  C:/ColdFusion9/runtime/../lib/commons-httpclient-3.1.jar;  C:/ColdFusion9/runtime/../lib/commons-lang-2.4.jar;  C:/ColdFusion9/runtime/../lib/commons-logging-1.1.1.jar;  C:/ColdFusion9/runtime/../lib/commons-logging-api-1.1.1.jar;  C:/ColdFusion9/runtime/../lib/commons-net-2.0.jar;  C:/ColdFusion9/runtime/../lib/commons-vfs-1.0.jar;  C:/ColdFusion9/runtime/../lib/crystal.jar;  C:/ColdFusion9/runtime/../lib/derby.jar;  C:/ColdFusion9/runtime/../lib/derbyclient.jar;  C:/ColdFusion9/runtime/../lib/derbynet.jar;  C:/ColdFusion9/runtime/../lib/derbyrun.jar;  C:/ColdFusion9/runtime/../lib/derbytools.jar;  C:/ColdFusion9/runtime/../lib/dom4j-1.6.1.jar;  C:/ColdFusion9/runtime/../lib/ehcache-web.jar;  C:/ColdFusion9/runtime/../lib/ehcache.jar;  C:/ColdFusion9/runtime/../lib/FCSj.jar;  C:/ColdFusion9/runtime/../lib/flashgateway.jar;  C:/ColdFusion9/runtime/../lib/flex-messaging-common.jar;  C:/ColdFusion9/runtime/../lib/flex-messaging-core.jar;  C:/ColdFusion9/runtime/../lib/flex-messaging-opt.jar;  C:/ColdFusion9/runtime/../lib/flex-messaging-proxy.jar;  C:/ColdFusion9/runtime/../lib/flex-messaging-remoting.jar;  C:/ColdFusion9/runtime/../lib/geronimo-stax-api_1.0_spec-1.0.jar;  C:/ColdFusion9/runtime/../lib/hibernate3.jar;  C:/ColdFusion9/runtime/../lib/httpclient.jar;  C:/ColdFusion9/runtime/../lib/ib6addonpatch.jar;  C:/ColdFusion9/runtime/../lib/ib6core.jar;  C:/ColdFusion9/runtime/../lib/ib6http.jar;  C:/ColdFusion9/runtime/../lib/ib6swing.jar;  C:/ColdFusion9/runtime/../lib/ib6util.jar;  C:/ColdFusion9/runtime/../lib/im.jar;  C:/ColdFusion9/runtime/../lib/iText.jar;  C:/ColdFusion9/runtime/../lib/iTextAsian.jar;  C:/ColdFusion9/runtime/../lib/izmado.jar;  C:/ColdFusion9/runtime/../lib/jai_codec.jar;  C:/ColdFusion9/runtime/../lib/jai_core.jar;  C:/ColdFusion9/runtime/../lib/jai_imageio.jar;  C:/ColdFusion9/runtime/../lib/jakarta-oro-2.0.6.jar;  C:/ColdFusion9/runtime/../lib/jakarta-slide-webdavlib-2.1.jar;  C:/ColdFusion9/runtime/../lib/java2wsdl.jar;  C:/ColdFusion9/runtime/../lib/jax-qname.jar;  C:/ColdFusion9/runtime/../lib/jaxb-api.jar;  C:/ColdFusion9/runtime/../lib/jaxb-impl.jar;  C:/ColdFusion9/runtime/../lib/jaxb-libs.jar;  C:/ColdFusion9/runtime/../lib/jaxb-xjc.jar;  C:/ColdFusion9/runtime/../lib/jaxrpc.jar;  C:/ColdFusion9/runtime/../lib/jdom-1.0.jar;  C:/ColdFusion9/runtime/../lib/jeb.jar;  C:/ColdFusion9/runtime/../lib/jintegra.jar;  C:/ColdFusion9/runtime/../lib/jnbcore.jar;  C:/ColdFusion9/runtime/../lib/jpedal.jar;  C:/ColdFusion9/runtime/../lib/js.jar;  C:/ColdFusion9/runtime/../lib/jsch-0.1.41m.jar;  C:/ColdFusion9/runtime/../lib/jsr107cache.jar;  C:/ColdFusion9/runtime/../lib/jutf7-0.9.0.jar;  C:/ColdFusion9/runtime/../lib/ldap.jar;  C:/ColdFusion9/runtime/../lib/ldapbp.jar;  C:/ColdFusion9/runtime/../lib/log4j-1.2.15.jar;  C:/ColdFusion9/runtime/../lib/lucene-analyzers.jar;  C:/ColdFusion9/runtime/../lib/lucene.jar;  C:/ColdFusion9/runtime/../lib/lucenedemo.jar;  C:/ColdFusion9/runtime/../lib/macromedia_drivers.jar;  C:/ColdFusion9/runtime/../lib/mail.jar;  C:/ColdFusion9/runtime/../lib/metadata-extractor-2.4.0-beta-1.jar;  C:/ColdFusion9/runtime/../lib/mlibwrapper_jai.jar;  C:/ColdFusion9/runtime/../lib/msapps.jar;  C:/ColdFusion9/runtime/../lib/mysql-connector-java-commercial-5.1.7-bin.jar;  C:/ColdFusion9/runtime/../lib/namespace.jar;  C:/ColdFusion9/runtime/../lib/nekohtml.jar;  C:/ColdFusion9/runtime/../lib/ooxml-schemas.jar;  C:/ColdFusion9/runtime/../lib/pdfencryption.jar;  C:/ColdFusion9/runtime/../lib/poi-contrib.jar;  C:/ColdFusion9/runtime/../lib/poi-ooxml.jar;  C:/ColdFusion9/runtime/../lib/poi-scratchpad.jar;  C:/ColdFusion9/runtime/../lib/poi.jar;  C:/ColdFusion9/runtime/../lib/portlet_20.jar;  C:/ColdFusion9/runtime/../lib/postgresql-8.3-604.jdbc3.jar;  C:/ColdFusion9/runtime/../lib/relaxngDatatype.jar;  C:/ColdFusion9/runtime/../lib/ri_generic.jar;  C:/ColdFusion9/runtime/../lib/rome-cf.jar;  C:/ColdFusion9/runtime/../lib/saaj.jar;  C:/ColdFusion9/runtime/../lib/slf4j-api-1.5.6.jar;  C:/ColdFusion9/runtime/../lib/slf4j-log4j12-1.5.6.jar;  C:/ColdFusion9/runtime/../lib/smack.jar;  C:/ColdFusion9/runtime/../lib/smpp.jar;  C:/ColdFusion9/runtime/../lib/STComm.jar;  C:/ColdFusion9/runtime/../lib/tools.jar;  C:/ColdFusion9/runtime/../lib/tt-bytecode.jar;  C:/ColdFusion9/runtime/../lib/vadmin.jar;  C:/ColdFusion9/runtime/../lib/verity.jar;  C:/ColdFusion9/runtime/../lib/vparametric.jar;  C:/ColdFusion9/runtime/../lib/vsearch.jar;  C:/ColdFusion9/runtime/../lib/wc50.jar;  C:/ColdFusion9/runtime/../lib/webchartsJava2D.jar;  C:/ColdFusion9/runtime/../lib/wsdl2java.jar;  C:/ColdFusion9/runtime/../lib/wsdl4j-1.5.1.jar;  C:/ColdFusion9/runtime/../lib/wsrp4j-commons-0.5-SNAPSHOT.jar;  C:/ColdFusion9/runtime/../lib/wsrp4j-producer.jar;  C:/ColdFusion9/runtime/../lib/xalan.jar;  C:/ColdFusion9/runtime/../lib/xercesImpl.jar;  C:/ColdFusion9/runtime/../lib/xml-apis.jar;  C:/ColdFusion9/runtime/../lib/xmlbeans-2.3.0.jar;  C:/ColdFusion9/runtime/../lib/xmpcore.jar;  C:/ColdFusion9/runtime/../lib/xsdlib.jar;  C:/ColdFusion9/runtime/../lib/;  C:/ColdFusion9/runtime/../gateway/lib/examples.jar;  C:/ColdFusion9/runtime/../gateway/lib/;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/flex/jars/cfgatewayadapter.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/flex/jars/concurrent.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/flex/jars/;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/batik-awt-util.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/batik-css.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/batik-ext.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/batik-transcoder.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/batik-util.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/commons-discovery.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/commons-logging.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/concurrent.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/flex.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/jakarta-oro-2.0.7.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/jcert.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/jnet.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/jsse.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/oscache.jar;  C:/ColdFusion9/runtime/../wwwroot/WEB-INF/cfform/jars/;
Java Class Path	C:\ColdFusion9\runtime\servers\lib; C:\ColdFusion9\runtime\servers\lib\jrun-patch.jar; C:\ColdFusion9\runtime\..\lib\macromedia_drivers.jar; C:\ColdFusion9\runtime\lib\cfmx_mbean.jar; C:\ColdFusion9\runtime\..\lib\oosdk\classes; C:\ColdFusion9\runtime\..\lib\oosdk\lib; C:\ColdFusion9\runtime\..\lib\oosdk\lib\juh.jar; C:\ColdFusion9\runtime\..\lib\oosdk\lib\jurt.jar; C:\ColdFusion9\runtime\..\lib\oosdk\lib\ridl.jar; C:\ColdFusion9\runtime\..\lib\oosdk\lib\unoil.jar; C:\ColdFusion9\runtime\lib; C:\ColdFusion9\runtime\lib\cfmx_mbean.jar; C:\ColdFusion9\runtime\lib\instutil.jar; C:\ColdFusion9\runtime\lib\java2wsdl.jar; C:\ColdFusion9\runtime\lib\jrun-ant-tasks.jar; C:\ColdFusion9\runtime\lib\jrun-xdoclet.jar; C:\ColdFusion9\runtime\lib\jrun.jar; C:\ColdFusion9\runtime\lib\jspc.jar; C:\ColdFusion9\runtime\lib\migrate.jar; C:\ColdFusion9\runtime\lib\oem-xdoclet.jar; C:\ColdFusion9\runtime\lib\sniffer.jar; C:\ColdFusion9\runtime\lib\webservices.jar; C:\ColdFusion9\runtime\lib\wsconfig.jar; C:\ColdFusion9\runtime\lib\wsdl2java.jar; C:\ColdFusion9\runtime\lib\xmlscript.jar; C:\ColdFusion9\runtime\lib\jrun.jar
Java Ext Dirs	C:\ColdFusion9\runtime\jre\lib\ext;C:\Windows\Sun\Java\lib\ext
Printer Details
Default Printer
Printers	Microsoft XPS Document Writer (redirected 2) Fax (redirected 2) HP Photosmart C7200 series (redirected 2) Microsoft XPS Document Writer

the minimum value for JVM heap size is blank.

Sorry, I did not understand. Are you suggesting

-server -Dsun.io.useCanonCaches=false -XX:PermSize=192m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

as a new string?

[carl type3]

Yes and place 512 in the "Minimum JVM Heap Size (MB)" as well.

>Are you suggesting

-server -Dsun.io.useCanonCaches=false -XX:PermSize=192m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch etc as a new string?

It would be fair to say CF9 is not "patched up" being 9.0.0 with no update level where as current being 9.0.2. CF9.0.0 could have updater1 or cumulative hotfix 2 applied or both plus some security fixes. Perhaps more noteworthy is the Java version. CF9.0.2 and CF10 both use 1.6.0_29 and 1.6.0_33 is current so you can see 1.6.0_14 is old. Having said all that I don't know if your CFM application even works beyond CF9.0.0 so you may not want to apply any CF9 updates or patches without performing acceptance testing. Read as I don't want you to say since Carl said use CF9.0.1 + CHF2 etc now my CFM application does not work.

Regards, Carl.

[Rohit S]

have you considered scheduled restarts for your CF service? a simple batch script can eliminate a lot of memory consumption issues

Rohit S