Copy link to clipboard
Copied
The last 2 days - my [removed link] CF site has suffered from high CPU usage (on and off) that stops functionality.
This will last several minutes then suddenly the high usage stops a few minute later the high cpu usage starts again.
I don't have any scheduled processes allowed to run.
Restarting the coldfusion service during a high CPS incident - the cpu usage just picks up where it left off when restarted.
I have assigned 4 CPUs (normal) or 8CPUs (as a test) to the VM and I get the same high CPU then low usage. This is a fresh install several months ago on a new VM.
I'd like some Adobe support on this issue but can't seem to find the contact phone number.
This series of high and low cpu usage first started Monday AM about midnight until 7:00 AM Tuesday was quiet and Wednesday this AM I'm having the same issues.
Ideas?
Update: While waiting for this to post - the high CPU usage started again and this time I stopped and restarted the Jetty service ... and the CPU usage went away...
Update: the high cpu started again and a restart of the jetty service didn't help --- neither did the .net service restart... the cf10 application service did stop the high usage...
Jay Bietz
Message was edited by: Jay Bietz after post accepted.
Copy link to clipboard
Copied
Hi Jay,
You have Enterprise so details gleaned from CF Monitor may help narrow down the cause of high CPU. Does CF Monitor - slowest active requests - show a particular program consuming resources? What does CF Monitor - JVM memory (used / max) MB - say about memory usage?
You could do well to know if the CF tomcat thread / pools are bottlenecking. Will need JDK tools like Java Mission Control and JMX enabled in CF JVM settings to get a look at the tomcat usage. Many times error details in ISAPI log (ColdFusion10\config\wsconfig\N) can give you a hint that CF tomcat parameters need adjusting.
HTH, Carl.
Copy link to clipboard
Copied
Carl:
Thanks for the ideas -- we found the offending and seldom used feature that when used was causing the massive CPU load on the server -- I'll research your suggestions as well as the original code and server to server communications. We've removed the feature until more is known and it's fixed --- had a quiet night since the feature was removed:)
"What does CF Monitor - JVM memory (used / max) MB - say about memory usage?" I'm seeing during an event yesterday over a 1hour 40 period JVM memory max 926.00 MB used memory 728.84MB.
I have been watching the CF moniter, serve performance moniter and cf log files -- saw the slow down but hard to isolate the cause. See below for illustration of a spike but how do I find out what caused it?
Thanks again and I'll advise as needed here.
Jay
Copy link to clipboard
Copied
I have the jconsole working on a devlopment server - will do the production setup next week ... will be remote from my workstation etc.
I searched the server for the Isapi log file but didn't find any hits - I don't have one in the ColdFusion10\config\wsconfig\N area do have isapi_redirect.dll and .properties... nothing that looks like a log.
Jay
Copy link to clipboard
Copied
Hi Jay,
Suggest check in "Requests that timed out" and "Requests slower than 20 seconds" for a CFc/m that took about 5 minutes. There might be more than one request because it appears all requests would have slowed in that interval from 10:30 to 10:35. So if you find a CFM that normally does not get a slow and did at this time ignore that and pick on one of the others to check on.
What does isapi_redirect.properties say about logging?
HTH again, Carl.
Copy link to clipboard
Copied
Carl:
Update - after a quite (no hung server alerts but some slow server emails) on Thursday -- 10/2 - Friday 10/3 had all the same slow CF messages and hung server (we have restart the CF application service or restart the Virtual Machine to clear error) .
Saturday and Sunday just the slow server alerts.
Odd that a workday Monday Wednesday and Friday - had the hung server but Tuesday, Thursday, Saturday and Sunday just the slow server emails.
Here is the isapi_redirect.properties but no log file in the folder.
extension_uri= /jakarta/isapi_redirect.dll
log_file= C:\ColdFusion10\config\wsconfig\1\isapi_redirect.log
log_level= info
worker_file= C:\ColdFusion10\config\wsconfig\1\workers.properties
worker_mount_file= C:\ColdFusion10\config\wsconfig\1\uriworkermap.properties
iis_buffer_enable= true
I'm looking at the slow server and requests timed out ...
TIA
Jay
Copy link to clipboard
Copied
Hi Jay,
I think the answers to the problem will mostly come from what you find out about the the slow and timed out requests.
Odd no log file even tho properties defines one; log_file= C:\ColdFusion10\config\wsconfig\1\isapi_redirect.log . Do not know why that would occur or even the results of not having a log. Perhaps nothing detrimental just less ability to know if there is a tomat thread pool issue happening.
While I think about the path C:\ColdFusion10\config\wsconfig\1\ what is date stamp of isapi_redirect.dll file? Some of the CF10 updates patched this however it is a manual step post update to apply change.
Regards, Carl.
Copy link to clipboard
Copied
Carl:
I've found and removed from service a second bit of code that sync'd cloud records with local records -- seems that last 8 days that there have been a large number of small changes - causing a large import of records in to the local DB's -- overwhelming the system.
I've changed JVM configs for -Xms and -Xmx from 512 to 4096 seemed to also help.
# Arguments to VM
java.args=-server -Xms4096m -Xmx4096m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion\\Verisign.jar
Still had a slow down and hang but only one yesterday... looking at logs today ...
Thanks
Jay
Copy link to clipboard
Copied
Hi Jay,
Making JVM adjustments can be a good idea. I think JVM changes are better with logging or JMX enabled so one can know wise choices are being made. I think the combination of JVM args selected are OK overall tho a setting for New Generation like -Xmn256m can be useful with those others you have set.
Still hangs so part of the problem could be tomcat. Without the tomcat connector log file you are not without tools to know how that is going on. You have JMX enabled and JDK Jconsole monitoring so you could do well to check how the tomcat thread AJP is doing. I discuss and demonstrate that here:
With regard to CF tomcat you might like this official CF blog:
http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning
HTH again, Carl.
Copy link to clipboard
Copied
After 2 hang free days -- Friday midnight (above is the gcviewer ) and ~2 hours later had the hang condition again. Current java.config includes
java.args=-server -Xms4096m -Xmx4096m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmn1024m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfwhispererGClive4.log -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion\\Verisign.jar
GCviewer shows that the full GCcollections suddenly start firing off many times in a short period of time.
Jay
will work on the tomcatlog issues next.
Copy link to clipboard
Copied
Thanks to Carl for his replies/ideas and especially Mike Brunt for consulting the last few days on the proper JVM.config settings - CF is running much better -- actually no pauses/hangs in the last 36 hours !!
Of course the fixes are numerous...
1. tightened up the firewall to limit unnecessary traffic to the site.
2. incomplete instructions for updating default install version of java to Java SE DK 1.7.0_60 server months ago. You need to move the JDKHOME/lib/tools.jar to in my case c:/cfusion10/cfusion/lib/ and replace the original version. Seems the tools.jar is only the JDK version and is a manual move after installing the update.
3. Many odd code errors were chased down and fixed -- and there are more to go.
4. A local piece of custom code that was supposed to send emails with more information to the programmers when a visitor had a page error - turned out to be a major hit on resources thus causing more errors and more emails etc.... some of 1M in size - many 300K in size.
Thanks again.
Jay Bietz