Intermittent Database Connecitivy Problems in CF10.
We have recently rolled out ColdFusion 10 Standard 64-bit to our production environment. We've installed the mandatory update as well as the two available updates. Our web server is IIS on Microsoft Server 2008 R2 64-bit.
The symptoms we are experiencing are:
- Loading ColdFusion pages that use database queries (CFQUERY) start to take a long time to load.
- In application.log we see several of the following errors
- "Error Executing Database Query.Timed out trying to establish connection"
- "The request has exceeded the allowable time limit"
- If we wait for the request to be served (that is we don't hit stop in the web browser) we eventually get one of the following errors:
- "[Macromedia][SequeLink JDBC Driver][ODBC Socket][Microsoft][ODBC Microsoft Access Driver] Too many client tasks."
- "Error Executing Database Query. Timed out trying to establish connection"
- On the most heavily trafficked pages we will see other errors that correspond to the time we experience this problem that look like:
- "Element RECORDCOUNT is undefined in <query name>"
- If we load a web page that performs non-database-querying operations (file reads, parsing text, whatever) the pages load fine and quickly even during the period of poor performance on pages with database operations.
---EDIT BEGIN---
We cache queries heavily.
During one period of trouble a query's cache time had expired and was rerun. In subsequent attempts to run this CACHED query the query variable, after the CFQUERY, does NOT exist according to CF. e.g.
<cfquery name="myQuery" ... cachedwithin="#CreateTimeSpan(0,0,30,0)#">
...
</cfquery>
<cfif IsDefined( "myQuery" ) is false> BAD QUERY </cfif>
This would output "bad query" even after the period of slowness had subsided. If I rerun the query without the cachedwithin attribute everything is fine and subsequent query operations, with or without cachedwithin set, are fine (because the cached copy has been revised).
This is why we're seeing those "Element <element name> is undefiend in <query name>" errors.
---EDIT END---
If we do nothing the server returns to its normal, responsive self in about 5 minutes.
The time between these five minute blocks of database issues can vary between 15 minutes and several hours.
Note that we use a lot of Access databases. I am aware that Access datasources should have the "maintain connections" left unchecked and they all do.
These symptoms, to me, seem to indicate there is a problem with ODBC resources, as if something clogs up the ODBC "pipe" and everything is stuck waiting for it to clear, but once it clears up everything is back to normal. I'm not sure of that's an ODBC Server/Agent issue or could it be an issue with how ColdFusion allocates resources (threads) to communicate with the ODBC Server/Agent.
Has anyone else experienced this and perhaps found a reason or even a resolution?
--BEGIN EDIT--
Our version of ColdFusion is 64-bit. I believe SequeLink is 32-bit. Could this possibly be related to the problem?
Can anyone offer some advice on how I could further troubleshoot the issue to narrow down the problem?
--END EDIT--
Any help is greatly appreciated.
Thanks
