Highlighted

CF 10 sp 12 Enterprise high CPU usage hangs services, 2012 server on 2012 Hyper-V host

Community Beginner ,
Oct 01, 2014

Copy link to clipboard

Copied

The last 2 days - my [removed link] CF site has suffered from high CPU usage (on and off) that stops functionality.

This will last several minutes then suddenly the high usage stops a few minute later the high cpu usage starts again.

I don't have any scheduled processes allowed to run.


Restarting the coldfusion service  during a high CPS incident - the cpu usage just picks up where it left off when restarted.

I have assigned 4 CPUs (normal) or 8CPUs  (as a test) to the VM and I get the same high CPU then low usage. This is a fresh install several months ago on a new VM.

I'd like some Adobe support on this issue but can't seem to find the contact phone number.

This series of high and low cpu usage first started Monday AM about midnight until 7:00 AM  Tuesday was quiet and Wednesday this AM I'm having the same issues.

Ideas?

heavy cpu usage4high.JPG

heavy cpu usage3normal.JPG

Update:  While waiting for this to post - the high CPU usage started again and this time I stopped and restarted the Jetty service ... and the CPU usage went away...

Update: the high cpu started again and a restart of the jetty service didn't help --- neither did the .net service restart... the cf10 application service did stop the high usage...

Jay Bietz

Message was edited by: Jay Bietz after post accepted.

Views

904

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

CF 10 sp 12 Enterprise high CPU usage hangs services, 2012 server on 2012 Hyper-V host

Community Beginner ,
Oct 01, 2014

Copy link to clipboard

Copied

The last 2 days - my [removed link] CF site has suffered from high CPU usage (on and off) that stops functionality.

This will last several minutes then suddenly the high usage stops a few minute later the high cpu usage starts again.

I don't have any scheduled processes allowed to run.


Restarting the coldfusion service  during a high CPS incident - the cpu usage just picks up where it left off when restarted.

I have assigned 4 CPUs (normal) or 8CPUs  (as a test) to the VM and I get the same high CPU then low usage. This is a fresh install several months ago on a new VM.

I'd like some Adobe support on this issue but can't seem to find the contact phone number.

This series of high and low cpu usage first started Monday AM about midnight until 7:00 AM  Tuesday was quiet and Wednesday this AM I'm having the same issues.

Ideas?

heavy cpu usage4high.JPG

heavy cpu usage3normal.JPG

Update:  While waiting for this to post - the high CPU usage started again and this time I stopped and restarted the Jetty service ... and the CPU usage went away...

Update: the high cpu started again and a restart of the jetty service didn't help --- neither did the .net service restart... the cf10 application service did stop the high usage...

Jay Bietz

Message was edited by: Jay Bietz after post accepted.

Views

905

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Oct 01, 2014 0
Guide ,
Oct 01, 2014

Copy link to clipboard

Copied

Hi Jay,

You have Enterprise so details gleaned from CF Monitor may help narrow down the cause of high CPU. Does CF Monitor - slowest active requests - show a particular program consuming resources? What does CF Monitor - JVM memory (used / max) MB - say about memory usage?

You could do well to know if the CF tomcat thread / pools are bottlenecking. Will need JDK tools like Java Mission Control and JMX enabled in CF JVM settings to get a look at the tomcat usage. Many times error details in ISAPI log (ColdFusion10\config\wsconfig\N) can give you a hint that CF tomcat parameters need adjusting.

HTH, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 01, 2014 0
Community Beginner ,
Oct 02, 2014

Copy link to clipboard

Copied

Carl:

Thanks for the ideas -- we found the offending and seldom used feature that when used was causing the massive CPU  load on the server  -- I'll research your suggestions as well as the original code and server to server communications.  We've removed the feature until more is known and it's fixed --- had a quiet night since the feature was removed:)

"What does CF Monitor - JVM memory (used / max) MB - say about memory usage?"  I'm seeing during an event yesterday over a 1hour 40 period JVM memory max 926.00 MB used memory 728.84MB.  

I have been watching the CF moniter, serve performance moniter and cf log files -- saw the slow down but hard to isolate the cause.  See below for illustration of a spike but how do I find out what caused it?

spike and pingdomehits..JPG

Thanks again and I'll advise as needed here.

Jay

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 02, 2014 0
Community Beginner ,
Oct 02, 2014

Copy link to clipboard

Copied

I have the jconsole working on a devlopment server - will do the production setup next week ... will be remote from my workstation etc.

I searched the server for the Isapi  log file but didn't find any hits - I don't have one in the ColdFusion10\config\wsconfig\N area do have isapi_redirect.dll and .properties... nothing that looks like a log.

Jay

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 02, 2014 0
Guide ,
Oct 02, 2014

Copy link to clipboard

Copied

Hi Jay,

Suggest check in "Requests that timed out" and "Requests slower than 20 seconds" for a CFc/m that took about 5 minutes. There might be more than one request because it appears all requests would have slowed in that interval  from 10:30 to 10:35. So if you find a CFM that normally does not get a slow and did at this time ignore that and pick on one of the others to check on.

What does isapi_redirect.properties say about logging?

HTH again, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 02, 2014 0
Community Beginner ,
Oct 05, 2014

Copy link to clipboard

Copied

Carl:

Update - after a quite (no hung server alerts but some slow server emails) on Thursday -- 10/2  - Friday 10/3  had all the same slow CF messages and hung server (we have  restart the CF application service or restart the Virtual Machine to clear error) .

Saturday and Sunday just the slow server alerts. 

Odd that a workday Monday Wednesday and Friday - had the hung server but Tuesday,  Thursday, Saturday and Sunday  just the slow server emails.

Here is the isapi_redirect.properties but  no log file in the folder. 

extension_uri= /jakarta/isapi_redirect.dll

log_file= C:\ColdFusion10\config\wsconfig\1\isapi_redirect.log

log_level= info

worker_file= C:\ColdFusion10\config\wsconfig\1\workers.properties

worker_mount_file= C:\ColdFusion10\config\wsconfig\1\uriworkermap.properties

iis_buffer_enable= true

I'm looking at the slow server and requests timed out ...

TIA
Jay

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 05, 2014 0
Guide ,
Oct 06, 2014

Copy link to clipboard

Copied

Hi Jay,

I think the answers to the problem will mostly come from what you find out about the the slow and timed out requests.

Odd no log file even tho properties defines one; log_file= C:\ColdFusion10\config\wsconfig\1\isapi_redirect.log . Do not know why that would occur or even the results of not having a log. Perhaps nothing detrimental just less ability to know if there is a tomat thread pool issue happening.

While I think about the path C:\ColdFusion10\config\wsconfig\1\ what is date stamp of isapi_redirect.dll file? Some of the CF10 updates patched this however it is a manual step post update to apply change.

Regards, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 06, 2014 0
Community Beginner ,
Oct 07, 2014

Copy link to clipboard

Copied

Carl:

I've found and removed from service a second bit of code that sync'd cloud records with local records -- seems that last 8 days that there have been a large number of small changes - causing a large import of records in to the local DB's  -- overwhelming the system.

I've changed JVM configs for -Xms and -Xmx from 512 to 4096  seemed to also help.

# Arguments to VM

java.args=-server  -Xms4096m -Xmx4096m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion\\Verisign.jar

Still had a slow down and hang but only one yesterday... looking at logs today ...

Thanks

Jay

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 07, 2014 0
Guide ,
Oct 07, 2014

Copy link to clipboard

Copied

Hi Jay,

Making JVM adjustments can be a good idea. I think JVM changes are better with logging or JMX enabled so one can know wise choices are being made. I think the combination of JVM args selected are OK overall tho a setting for New Generation like -Xmn256m can be useful with those others you have set.

Still hangs so part of the problem could be tomcat. Without the tomcat connector log file you are not without tools to know how that is going on. You have JMX enabled and JDK Jconsole monitoring so you could do well to check how the tomcat thread AJP is doing. I discuss and demonstrate that here:

http://vimeo.com/67571552

With regard to CF tomcat you might like this official CF blog:

http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning


HTH again, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 07, 2014 0
Community Beginner ,
Oct 11, 2014

Copy link to clipboard

Copied

1010hange.JPG

After 2 hang free days -- Friday midnight (above is the gcviewer )  and ~2 hours later had the hang condition again.  Current java.config includes

java.args=-server  -Xms4096m -Xmx4096m -Xdebug -Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5005 -XX:PermSize=512m -XX:MaxPermSize=512m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false -Xmn1024m -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfwhispererGClive4.log -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion,C:\\CFusion10\\cfusion\\verisign\\payflowpro\\Java\\PFProColdFusion\\Verisign.jar

GCviewer shows that the full GCcollections suddenly start firing off many times in a short period of time.

Jay

will work on the tomcatlog issues next.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 11, 2014 0
Community Beginner ,
Oct 30, 2014

Copy link to clipboard

Copied

201410300933.JPG

Thanks to Carl for his replies/ideas and especially Mike Brunt for consulting the last few days on the proper JVM.config settings - CF is running much better -- actually no pauses/hangs in the last 36 hours !!

Of course the fixes are numerous...

1. tightened up the firewall to limit unnecessary traffic to the site. 

2. incomplete instructions for updating default install version of java to Java SE DK 1.7.0_60 server months ago. You need to move the JDKHOME/lib/tools.jar to in my case c:/cfusion10/cfusion/lib/  and replace the original version.  Seems the tools.jar is only the JDK version and is  a manual move after installing the update.

3. Many odd code errors were chased down and fixed -- and there are more to go.

4. A local piece of  custom code that was supposed to send emails with more information to the programmers  when a visitor had a page error - turned out to be a major hit on resources thus causing more errors and more emails etc.... some of 1M in size - many 300K in size.

Thanks again.

Jay Bietz

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Oct 30, 2014 2