CF21U12 - SQL DSN stopped working - Connection verification failed for data source

[mguenther1272]

Hey all.

Had something weird happened last night/this morning.

Post OS Updates (I don't think it's MS Updates - but it is a change) last night my production instance of our CF21 server stopped connecting to the backend SQL Server with the following error:

Connection verification failed for data source: <DSNName>
java.sql.SQLException: Timed out trying to establish connection

Looking in the coldfusion-error.log I see....

Exception in thread "Thread-948" java.lang.NoClassDefFoundError: com/ddtek/jdbc/extensions/ExtEmbeddedConnection
at macromedia.jdbc.sqlserver.base.r.dg(|SQLServer|6.0.0.1282|:49)
at macromedia.jdbc.sqlserver.base.s.I(|SQLServer|6.0.0.1282|:135)
at macromedia.jdbc.sqlserver.base.BaseDriver.connect(|SQLServer|6.0.0.1282|:190)
at macromedia.jdbc.MacromediaDriver.connect(MacromediaDriver.java:256)
at coldfusion.server.j2ee.sql.pool.JDBCPool.createPhysicalConnection(JDBCPool.java:666)
at coldfusion.server.j2ee.sql.pool.ConnectionRunner$RunnableConnection.run(ConnectionRunner.java:67)
at java.base/java.lang.Thread.run(Thread.java:834)
A non-SQL error occurred while requesting a connection from <DSNName>.
Timed out trying to establish connection

I've tried deleting and recreating the DSN - same result.

I've verified that the CF Server DOES have network connectivity over 1433 to the PRD SQL Server using Test-NetConnection <servername> -port 1433 and everything works there.

I can connect to the SQL server using SSMS - no issues there.

I can connect to the PRD SQL server from our DEV server (which got the OS updates last week with no issues) - so no issues there either.

I had this same issue crop up on another CF server (running Enterprise) a couple of weeks ago - that one I fixed by simply creating a new instance on that server from scratch - but this is a CF Standard install, and I think I'd have to uninstall and reinstall CF to effectively do that here, and while I could do that - it'd take more time than I'd want (since I'm technically on vacation 8) ).

[Charlie Arehart]

I'll offer some guesses, first to help you possibly diagnose the problem, then to possibly cure it (many folks toss out the latter without offering the former. That makes for briefer answers, but they're then just darts to throw at a problem--and some darts can do unexpected damage if not used carefully).

Let's see if these may help you:

1) I don't think that what triggered things was the os update. Instead, that forced a restart of cf, and I'd suspect something had changed about cf since the last time cf had been restarted

• There's a log called server.log in the cf logs folder.. Can you look at it and report when your prior cf restart was? That will help you focus on what may have changed since then. (Yes, such cf restarts are tracked also in other files, but often with a lot more info that makes it hard to find the restarts--and they may fill and rotate)

2) This problem sounds like it's related to the cf package mgt aspect new to cf2021 and above. And it would seem related specifically to the sql server package. 

• If you look at the startup messages in the cf logs (coldfusion-out, coldfusion-error, and/or cfpm-audit), you may see some issue related to this. 
• After an update, you will often see package management log entries reflecting the package update process during the startup. 
• If you see any errors related to package mgt during startup, what do they say?
• As an aside, does this server have internet access or is it offline, air-gapped, or is outbound access tightly controlled? 

3) Rather than reinstall cf, or create a new instance, what happens if you go to the package mgt page of the cf admin and just uninstall them reinstall the sql server module?

• Some will want to note that there is also the command line cfpm tool, in the cfusion/bin folder. That's an option, especially if somehow the admin was unavailable.

4) Some problems with packages/modules are of a sort that Adobe recommends simply "blowing away" the folder that holds the deployed packages, which is the felix-cache folder within the cfusion/bin folder. 

• You could stop cf and a) delete all files in it or, b) delete the folder. Cf would recreate it on startup. If you were hesitant about deleting, you could instead c) rename the folder or d) move it elsewhere. On restart x cf will create the folder and redeploy the packages (which are already downloaded and in the bundles folder within the root of the cf folder). 

Lots to consider, yes. Some diagnostics, some procedures. I could have offered only the latter. I never know if readers are fellow doctors (wanting to understand things) or patients (just willing to take any medicine if it might help). And some problems are new enough that there's not yet a wide body of known knowledge--let alone concensus--on solving certain problems. I'm hoping to contribute to the knowledge base, as well as to learn if there are counterpoints I've not considered. 

So please let us know what you find or how it goes. And hope after this you can get back to enjoying your vacation. 🙂 

[mguenther1272 · Accepted solution]

Charlie,

Thanks for the (as always) thorough and engaging reply 8)  You're a treasure 8)

I submitted my symptoms to Adobe Support and they got back to me a short while ago.  Essentially their reply boiled down to, that it's a known bug and they're working on a fix.

They provided me with a work around:

1. Stop CF services and delete the contents of <CFROOT>\cfusion\bin\felix-cache

2. delete .CLASS files in <CFROOT>\cfusion\wwwroot\WEB-INF\cfclasses

3. restart CF Services

Sure enough that fixed the problem.

You're probably right about the package management thing being the root of the problem.  I remember earlier this month when I updated to Update 12 - I essentially had to use cfpm to uninstall all the packages and reinstall them because nothing was working after applying U12 - I had to do that for all of my instances - it was really weird.

I'd be interested in figuring things out further - if this hadn't been our production system, and the University wasn't about to go on our annual winter closing 8).

As much as I've loved working as a CF Admin these past 14 years - I can't wait for our developers to wrap up their .NET conversions so that we can migrate our apps into Azure.... (1.5 down and 6.5 to go....)

[Charlie Arehart]

Thanks for the update, and the kind regards. As for their offered solution, their step 1 was indeed my second proposed one (in my item 4), of that wasn't clear. But what matters is you're issue is sorted out...and this post and both answers will stand for others who may have the same problem and might find this thread.