CPU Usage going crazy

[matthisco]

Hi Folks,

My CPU usage is at 100% and the webpages are loading without the stylesheet.

I've restarted a few services and it seems to be better, but the CPU usage is still a bit erratic.

Can anyone please tell me how I can find out what the problem is? Is there a fault finding process I can go through, restarting each service one at a time?

Coldfusion 7/ windows server 2003/ IIS v6

Many thanks

[carl type3]

Hi,

It would be good to confirm it is jrun.exe that is the process cpu bound and not something else taking the cpu usage?

If jrun.exe then start by having a look thru the CFadminUI > Debugging & Logging > Log Files. Also check CF\runtime\logs for errors.

HTH, Carl.

[Charlie Arehart]

Matt, while some would immediately wonder if you have "some long running looping page", in my experience the high cpu is more typically reflective of instead a JVM memory issue. (And it may not be a "you need to add more heap" error. It may instead be a "you need to add more maxpermsize", or even "you need to have less heap", believe it or not.)

Rather than speculate, I offer details on how to start sorting such things out by showing you where to look and how to read the tea leaves at this blog entry:

CF911: Lies, damned lies, and when memory problems not be at all what they seem, Part 1

http://www.carehart.org/blog/client/index.cfm/2010/11/3/when_memory_problems_arent_what_they_seem_part_1

I know I still need to get to the other parts I outlined, but even without them, the introductory diagnostic info may help you.

Let us know how it goes.

/charlie arehart

charlie@carehart.org

Providing CF and CFBuilder troubleshooting services

at http://www.carehart.org/consulting

[matthisco]

Thanks very much for your replies, really appreciate it.

I've gone back through the error logs.

I did get a java error just before everything stated to go belly up:

"An unhandled win32 exception occurred in java.exe [1864]. Just-In-Time debugging this exception failed with the following error: Debugger could not be started because no user is logged on."

However, I did get loads of these warnings throughout the night, do you think they might have anything to do with it?

"Failed to delete the queued snapshot(sqlinfo__xxx__xxx_deleting) on Storage System(xxxxx) volume(xxxxx).

Error code : A FlexClone based on the snapshot still exists or SnapMirror replication is in progress."

Thanks again for your replies

[carl type3]

Hi,

The unhandled java exception error that happens before the server “goes crazy” is probably the best indicator of the issue. Other errors occurring after that are probably symptomatic of the first error. I still think the details before and during the time when java is cpu bound in CFusionMX7\runtime\logs\ coldfusion-out.log (JRun4\logs\ in case of multi-server) will provide more idea as to what is happening. Perhaps you could paste a sample?

Not seen those “nightly” error messages so I got no idea on them perhaps others here can comment more.

ColdFusion 7 – What java virtual machine version is CF using? Is CF7 updater 2 applied?

HTH, Carl.

[matthisco]

THanks very much for your reply again. I've included a few lines of the runtime log, I remoted in at around 21:30 the night before and everything seemed fine, so I gather it must of been something overnight or first thing. The service starts at 07:47 are from restarting the server. Anymore ideas very much appreciated, thanks again.

Java details:

Java Version     1.4.2_09 
Java Vendor     Sun Microsystems Inc. 
Java Vendor URL     http://java.sun.com/ 
Java Home     C:\CFusionMX7\runtime\jre 
Java File Encoding     Cp1252 
Java Default Locale     en_GB 
File Separator     \ 
Path Separator     ; 
Line Separator     Chr(13) 
User Name     SYSTEM 
User Home     C:\Documents and Settings\Default User 
User Dir     C:\CFusionMX7\runtime\bin 
Java VM Specification Version     1.0 
Java VM Specification Vendor     Sun Microsystems Inc. 
Java VM Specification Name     Java Virtual Machine Specification 
Java VM Version     1.4.2_09-b05 
Java VM Vendor     Sun Microsystems Inc. 
Java VM Name     Java HotSpot(TM) Server VM 
Java Specification Version     1.4 
Java Specification Vendor     Sun Microsystems Inc. 
Java Specification Name     Java Platform API Specification 
Java Class Version     48.0 

Runtime details:

02/09 14:20:36 Information [scheduler-4] - Run Client Storage Purge

02/09 15:27:44 Information [scheduler-0] - Run Client Storage Purge

02/09 16:34:53 Information [scheduler-4] - Run Client Storage Purge

02/09 17:42:01 Information [scheduler-1] - Run Client Storage Purge

02/09 18:49:12 Information [scheduler-2] - Run Client Storage Purge

02/09 19:56:21 Information [scheduler-3] - Run Client Storage Purge

02/09 21:03:29 Information [scheduler-2] - Run Client Storage Purge

02/09 22:10:45 Information [scheduler-0] - Run Client Storage Purge

02/09 23:18:14 Information [scheduler-4] - Run Client Storage Purge

02/10 00:00:00 Information [scheduler-1] - [docindex] Executing at Thu Feb 10 00:00:00 GMT 2011

02/10 00:16:41 Information [scheduler-1] - [docindex] Rescheduling for :Fri Feb 11 00:00:00 GMT 2011 Now: Thu Feb 10 00:16:41 GMT 2011

02/10 00:19:37 Error [jrpp-3209] - The request has exceeded the allowable time limit Tag: cfoutput The specific sequence of files included or processed is: C:\Inetpub\wwwroot\tasks\docindex.cfm, line: 3302/10 00:25:22 Information [scheduler-4] - Run Client Storage Purge02/10 01:32:31 Information [scheduler-2] - Run Client Storage Purge02/10 02:39:38 Information [scheduler-1] - Run Client Storage Purge02/10 03:46:46 Information [scheduler-1] - Run Client Storage Purge02/10 04:53:54 Information [scheduler-1] - Run Client Storage Purge02/10 06:01:01 Information [scheduler-2] - Run Client Storage Purge02/10 07:08:10 Information [scheduler-1] - Run Client Storage PurgeStarting Macromedia JRun 4.0 (Build 92909), coldfusion server10/02 07:47:27 warning Unable to open C:\CFusionMX7\runtime/lib/license.properties10/02 07:47:29 info JRun Naming Service listening on *:292010/02 07:47:30 warning No sessionSecret has been specified in jrun.xml. Installing a self generated sessionSecret.10/02 07:47:32 info No JDBC data sources have been configured for this server (see jrun-resources.xml)10/02 07:47:32 info JRun Proxy Server listening on *:5101110/02 07:47:32 info Deploying enterprise application "Macromedia ColdFusion MX" from: file:/C:/CFusionMX7/10/02 07:47:33 info Deploying web application "Macromedia Coldfusion MX" from: file:/C:/CFusionMX7/10/02 07:47:40 INFO License Service: Flex 1.5 CF Edition enabled10/02 07:47:40 INFO Starting Flex 1.5 CF Edition10/02 07:47:40 user JSPServlet: init10/02 07:47:41 user MessageBrokerServlet: init10/02 07:47:44 user CFMxmlServlet: init10/02 07:47:44 user CFMxmlServlet: Macromedia Flex Build: 87315.13464610/02 07:47:44 INFO Macromedia Flex Build: 87315.13464610/02 07:47:49 user ColdFusionStartUpServlet: init10/02 07:47:49 user ColdFusionStartUpServlet: ColdFusion MX: Starting application services10/02 07:47:49 user ColdFusionStartUpServlet: ColdFusion MX: VM version = 1.4.2_09-b0502/10 07:47:49 Information [main] - Starting logging...02/10 07:47:50 Information [main] - Starting crypto...02/10 07:47:55 Information [main] - Starting license...02/10 07:47:55 Information [main] - Starting License server ...02/10 07:47:57 Information [main] - Starting scheduler...02/10 07:47:57 Information [main] - Starting WatchService...02/10 07:47:57 Information [main] - Starting debugging...02/10 07:47:57 Information [main] - Starting sql...02/10 07:48:38 Information [main] - Pool Manager Started02/10 07:48:38 Information [main] - Starting mail...02/10 07:48:40 Information [main] - CORBA Configuration not enabled02/10 07:48:40 Information [main] - Starting cron...02/10 07:48:44 Information [main] - [Joblog Week] Activating2 on Thu Feb 10 07:48:44 GMT 2011 To run on Mon Feb 14 06:00:00 GMT 201102/10 07:48:45 Information [main] - [docindex] Activating2 on Thu Feb 10 07:48:45 GMT 2011 To run on Fri Feb 11 00:00:00 GMT 201102/10 07:48:45 Information [main] - [month count] Activating2 on Thu Feb 10 07:48:45 GMT 2011 To run on Tue Mar 01 00:00:00 GMT 201102/10 07:48:45 Information [main] - Starting registry...02/10 07:48:47 Information [main] - Starting client...02/10 07:48:47 Information [main] - Starting xmlrpc...02/10 07:48:54 Information [main] - Starting graphing...02/10 07:49:08 Information [main] - Starting verity...02/10 07:49:08 Information [main] - Starting archive...02/10 07:49:08 Information [main] - Starting document...02/10 07:49:10 Information [main] - Starting eventgateway...02/10 07:49:10 Information [main] - Event Gateway DisabledRMI Registry started and listening on port 109902/10 07:49:10 Information [main] - Starting FlexAssembler...02/10 07:49:10 Information [main] - ColdFusion started10/02 07:49:10 user ColdFusionStartUpServlet: ColdFusion MX: application services are now available10/02 07:49:11 user CFSwfServlet: init10/02 07:49:11 user CFCServlet: init10/02 07:49:12 user FlashGateway: init10/02 07:49:12 user CFFormGateway: init10/02 07:49:12 user CFInternalServlet: initServer coldfusion ready (startup time: 111 seconds)02/10 07:54:59 Error [jrpp-0] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]Connection reset by peer: socket write error The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 42removeOnExceptions is true for sitecms, closed the physical ConnectionStarting Macromedia JRun 4.0 (Build 92909), coldfusion server10/02 08:25:09 warning Unable to open C:\CFusionMX7\runtime/lib/license.properties10/02 08:25:09 info JRun Naming Service listening on *:292010/02 08:25:10 warning No sessionSecret has been specified in jrun.xml. Installing a self generated sessionSecret.10/02 08:25:10 info No JDBC data sources have been configured for this server (see jrun-resources.xml)10/02 08:25:10 info JRun Proxy Server listening on *:5101110/02 08:25:10 info Deploying enterprise application "Macromedia ColdFusion MX" from: file:/C:/CFusionMX7/10/02 08:25:10 info Deploying web application "Macromedia Coldfusion MX" from: file:/C:/CFusionMX7/10/02 08:25:12 INFO License Service: Flex 1.5 CF Edition enabled10/02 08:25:12 INFO Starting Flex 1.5 CF Edition10/02 08:25:12 user JSPServlet: init10/02 08:25:12 user MessageBrokerServlet: init10/02 08:25:13 user CFMxmlServlet: init10/02 08:25:13 user CFMxmlServlet: Macromedia Flex Build: 87315.13464610/02 08:25:13 INFO Macromedia Flex Build: 87315.13464610/02 08:25:14 user ColdFusionStartUpServlet: init10/02 08:25:14 user ColdFusionStartUpServlet: ColdFusion MX: Starting application services10/02 08:25:14 user ColdFusionStartUpServlet: ColdFusion MX: VM version = 1.4.2_09-b0502/10 08:25:14 Information [main] - Starting logging...02/10 08:25:14 Information [main] - Starting crypto...02/10 08:25:15 Information [main] - Starting license...02/10 08:25:15 Information [main] - Starting License server ...02/10 08:25:16 Information [main] - Starting scheduler...02/10 08:25:16 Information [main] - Starting WatchService...02/10 08:25:16 Information [main] - Starting debugging...02/10 08:25:16 Information [main] - Starting sql...02/10 08:25:18 Information [main] - Pool Manager Started02/10 08:25:19 Information [main] - Starting mail...02/10 08:25:19 Information [main] - CORBA Configuration not enabled02/10 08:25:19 Information [main] - Starting cron...02/10 08:25:19 Information [main] - [Joblog Week] Activating2 on Thu Feb 10 08:25:19 GMT 2011 To run on Mon Feb 14 06:00:00 GMT 201102/10 08:25:19 Information [main] - [docindex] Activating2 on Thu Feb 10 08:25:19 GMT 2011 To run on Fri Feb 11 00:00:00 GMT 201102/10 08:25:19 Information [main] - [month count] Activating2 on Thu Feb 10 08:25:19 GMT 2011 To run on Tue Mar 01 00:00:00 GMT 201102/10 08:25:19 Information [main] - Starting registry...02/10 08:25:19 Information [main] - Starting client...02/10 08:25:19 Information [main] - Starting xmlrpc...02/10 08:25:20 Information [main] - Starting graphing...02/10 08:25:20 Information [main] - Starting verity...02/10 08:25:20 Information [main] - Starting archive...02/10 08:25:20 Information [main] - Starting document...02/10 08:25:21 Information [main] - Starting eventgateway...02/10 08:25:21 Information [main] - Event Gateway DisabledRMI Registry started and listening on port 109902/10 08:25:21 Information [main] - Starting FlexAssembler...02/10 08:25:21 Information [main] - ColdFusion started10/02 08:25:21 user ColdFusionStartUpServlet: ColdFusion MX: application services are now available10/02 08:25:21 user CFSwfServlet: init10/02 08:25:21 user CFCServlet: init10/02 08:25:21 user FlashGateway: init10/02 08:25:21 user CFFormGateway: init10/02 08:25:21 user CFInternalServlet: initServer coldfusion ready (startup time: 15 seconds)10/02 09:21:02 user CFMxmlServlet: destroy10/02 09:21:02 user CFCServlet: destroy10/02 09:21:02 user CFFormGateway: destroy10/02 09:21:02 user CFSwfServlet: destroy10/02 09:21:02 user FlashGateway: destroy10/02 09:21:02 user CFInternalServlet: destroy02/10 09:21:02 Information [Thread-0] - ColdFusion stoppedStarting Macromedia JRun 4.0 (Build 92909), coldfusion server10/02 09:21:09 warning Unable to open C:\CFusionMX7\runtime/lib/license.properties10/02 09:21:11 info JRun Naming Service listening on *:292010/02 09:21:11 warning No sessionSecret has been specified in jrun.xml. Installing a self generated sessionSecret.10/02 09:21:12 info No JDBC data sources have been configured for this server (see jrun-resources.xml)10/02 09:21:12 info JRun Proxy Server listening on *:5101110/02 09:21:12 info Deploying enterprise application "Macromedia ColdFusion MX" from: file:/C:/CFusionMX7/10/02 09:21:12 info Deploying web application "Macromedia Coldfusion MX" from: file:/C:/CFusionMX7/10/02 09:21:13 INFO License Service: Flex 1.5 CF Edition enabled10/02 09:21:13 INFO Starting Flex 1.5 CF Edition10/02 09:21:13 user JSPServlet: init10/02 09:21:14 user MessageBrokerServlet: init10/02 09:21:16 user CFMxmlServlet: init10/02 09:21:16 user CFMxmlServlet: Macromedia Flex Build: 87315.13464610/02 09:21:16 INFO Macromedia Flex Build: 87315.13464610/02 09:21:19 user ColdFusionStartUpServlet: init10/02 09:21:19 user ColdFusionStartUpServlet: ColdFusion MX: Starting application services10/02 09:21:19 user ColdFusionStartUpServlet: ColdFusion MX: VM version = 1.4.2_09-b0502/10 09:21:19 Information [main] - Starting logging...02/10 09:21:19 Information [main] - Starting crypto...02/10 09:21:20 Information [main] - Starting license...02/10 09:21:20 Information [main] - Starting License server ...02/10 09:21:20 Information [main] - Starting scheduler...02/10 09:21:20 Information [main] - Starting WatchService...02/10 09:21:20 Information [main] - Starting debugging...02/10 09:21:21 Information [main] - Starting sql...02/10 09:21:25 Information [main] - Pool Manager Started02/10 09:21:26 Information [main] - Starting mail...02/10 09:21:26 Information [main] - CORBA Configuration not enabled02/10 09:21:26 Information [main] - Starting cron...02/10 09:21:26 Information [main] - [Joblog Week] Activating2 on Thu Feb 10 09:21:26 GMT 2011 To run on Mon Feb 14 06:00:00 GMT 201102/10 09:21:26 Information [main] - [docindex] Activating2 on Thu Feb 10 09:21:26 GMT 2011 To run on Fri Feb 11 00:00:00 GMT 201102/10 09:21:26 Information [main] - [month count] Activating2 on Thu Feb 10 09:21:26 GMT 2011 To run on Tue Mar 01 00:00:00 GMT 201102/10 09:21:26 Information [main] - Starting registry...02/10 09:21:26 Information [main] - Starting client...02/10 09:21:26 Information [main] - Starting xmlrpc...02/10 09:21:26 Information [main] - Starting graphing...02/10 09:21:26 Information [main] - Starting verity...02/10 09:21:26 Information [main] - Starting archive...02/10 09:21:26 Information [main] - Starting document...02/10 09:21:27 Information [main] - Starting eventgateway...02/10 09:21:27 Information [main] - Event Gateway DisabledRMI Registry started and listening on port 109902/10 09:21:27 Information [main] - Starting FlexAssembler...02/10 09:21:27 Information [main] - ColdFusion started10/02 09:21:27 user ColdFusionStartUpServlet: ColdFusion MX: application services are now available10/02 09:21:27 user CFSwfServlet: init10/02 09:21:27 user CFCServlet: init10/02 09:21:27 user FlashGateway: init10/02 09:21:27 user CFFormGateway: init10/02 09:21:28 user CFInternalServlet: initServer coldfusion ready (startup time: 20 seconds)02/10 09:29:45 Error [jrpp-12] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]Connection reset by peer: socket write error The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 3002/10 09:29:45 Error [jrpp-1] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]Connection reset by peer: socket write error The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 3502/10 09:29:45 Error [jrpp-15] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]A problem occurred when attempting to contact the server (Server returned: Connection reset). Please ensure that the server parameters passed to the driver are correct and that the server is running. Also ensure that the maximum number of connections have not been exceeded for this server. The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 65removeOnExceptions is true for calendar, closed the physical ConnectionremoveOnExceptions is true for news, closed the physical ConnectionremoveOnExceptions is true for announcements, closed the physical ConnectionremoveOnExceptions is true for sitecms, closed the physical ConnectionremoveOnExceptions is true for vacancies, closed the physical ConnectionremoveOnExceptions is true for calendar, closed the physical Connection02/10 09:29:46 Error [jrpp-29] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]A problem occurred when attempting to contact the server (Server returned: Connection reset). Please ensure that the server parameters passed to the driver are correct and that the server is running. Also ensure that the maximum number of connections have not been exceeded for this server. The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 30removeOnExceptions is true for news, closed the physical ConnectionremoveOnExceptions is true for announcements, closed the physical ConnectionremoveOnExceptions is true for sitecms, closed the physical ConnectionremoveOnExceptions is true for vacancies, closed the physical Connection02/10 09:29:46 Error [jrpp-39] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]Connection reset by peer: socket write error The specific sequence of files included or processed is: C:\Inetpub\wwwroot\index.cfm, line: 2402/10 09:29:46 Error [jrpp-13] - Error Executing Database Query.[Macromedia][SQLServer JDBC Driver]A problem occurred when attempting to contact the server (Server returned: Connection reset). Please ensure that the server parameters passed to the driver are correct and that the server is running. Also ensure that the maximum number of connections have not been exceeded for this server. The specific sequence of files included or processed is: C:\Inetpub\wwwroot\portal.cfm, line: 38

[JR__Bob__Dobbs]

A couple of items to consider.

1. Your Java runtime should probably be updated to the latest version supported by CF7.  I believe this would be JDK version 1.4.2_15.  This may not fix your CPU issues but will address time zone related issues.
See: http://kb2.adobe.com/cps/547/2d547983.html and http://kb2.adobe.com/cps/447/d2ab4470.html

2. Your database related exceptions may be due to a communication failure between your CF server and database server.  I would verify that your SQL server allows TCP/IP connections and that the TCP/IP port is not being blocked by a firewall.

[carl type3]

+1 to what Bob said Java version and check traffic. For traffic sometimes the old PING command is useful perhaps use PING with the same packet size as your database is set to and or datasource is set ( -t to repeat -l for size / length)

HTH, Carl.