2021 Lockdown Installer nukes SQL Server Data Sources - Solved but root cause unknown. Any thoughts?

[PatrickHolway]

Morning folks,

So in my wonderful time these last few days building CF2021 servers on windows server 2022 Standard I have encountered an intersting issue that is consistently happening with each build.

I have built 4 servers and my process is as folows (brief version)

1) Build, patch, onboard and join the new server to our domain 

2) Install IIS with requisite Application Server components

3) Install CF 2021 using the GUI installer. The one I have comes with update 5 I believe , despite downloading a new from the adobe licensing site (maybe I'm missing soemthing but anyway)

4) Confirm CF Installed and service is running.

5) Update the JVM to open jdk 11-21 - ive had no issues running this. Restart CF of course

5) Download update 12 from package manager and install it by running \ColdFusion2021\jre\bin\java -Djdk.util.zip.disableZip64ExtraFieldValidation=true -jar C:\ColdFusion2021\cfusion\hf-updates\hotfix-012-330257.jar rather than the Download and Install Option as I find this is more reliable.

6) Comfrim I am on update 12 after a service restart and deploy a car archive from the current prod server which contains data sources and other good stuff.

7) Validate my data sources. They work fine.

8) Setup my 2 IIS sites 

9) run wsconfig.exe as an admin using default settings.

10) Check my sites work as expected and connect to my 2 sql server data sources. work flawlessly.

11) Validate the data sources in CF admin once again for good measure. Works.

12) run the CF 2021 auto lockdown installer with pretty much default settings. Installer completes with no errors.

13) Check my sites. No joy. Validate my data sources in CF Admin. No joy. Immediate eror message that the connection timed out. Add a new datasource suing the same details. no luck.

14) using Net Connect , I test the connection to the sql datasource. Works fine. Connect via powersell using Invoke-SQLCmd from the CF server. Works fine.

15 ) Check logs - only entries are minimal , cant connect, connection timed out.

16) check lockdown installer logs, no errors to speak of no fatals no severes, nada.

17 ) Ok check that SQL Server package is installed. It is

18 ) run cfpm.bat uninstall sqlserver, let it finsih then run cfpm.bat install sqlserver. restart coldfusion.

19) check data sources. All is well again. 🙂 

I am assuming the auto lockdown corrupts the package maybe? Anybody experienced this? Is my process out of order maybe? 

Hope you are all having a nice thursday. 

Thanks

Patrick

[PatrickHolway]

Quick note, I did not run the lockdown installer as an Administrator. Wondering if that might be the issue. Im gonna test the theory on my next box

[PatrickHolway]

Well running as an administrator did not solve the issue

[sdsinc_pmascari]

I have never encountered this.  Just a few thoughts....

Is the neo-datasource.xml file in the 'cfusion/lib' directory emptied?  Is there a neo-datasource.bak file present?  If so, can you restore the datasources with the backup file?

Otherwise, can you copy the neo-datasource.xml file to a temp directory before the lockdown install and then copy it back into the 'cfusion/lib' folder afterwards?

[Charlie Arehart]

(FWIW, I was writing my tome below while I guess Paul's here came in--and yep, his suggestion may well be at least the key way to recover, as I'd noted also, though I offered more to consider. Thanks for chiming in, Paul. I try to remember to check before submitting, when I've written a long note, in case someone else's answer comes in and would make me modify mine to acknowledge/consider theirs. I forgot to this time.) 

[Charlie Arehart]

First, thanks for all the details. Besides it helping us perhaps help you, it may help others contemplating the same goal.

As for your challenge, I have a few thoughts for you.

1) While you suspect the lockdown tool to be the issue, I'll propose at least one different possible culprit to rule out first. After you did the car import, while your DSN's "worked", it would be wise to do a CF restart.  I've seen situations where the import unexpectedly messed things up--in such a way that would not be clear until a CF restart. It would just be wise to try that as a sanity check in your steps; it just takes another minute or two.

But assuming the DSN's ARE still there after that, what's next to consider?

2) Well, you mention that you had the latest avaialble CF2021 installer (with update 5), and I'll add that in fact there WAS another updated installer released in Oct (for both CF2021, with its update 11, and CF2023, with its update 6). I see now that while I tweeted the news, I never blogged it. Bummer (for me, I mean). But Adobe did also mention it in passing in their thread about the CF updates released that day also. I don't think having that latest installer will be "the issue" for you, but I just wanted to offer that for you/others.

3) Perhaps more important, you don't say if you are using the latest version of the auto lockdown tool. Adobe did mention in that same post how they'd also updated that tool. Perhaps you're facing an issue addressed in that update (though there are never release notes for it that I've ever seen). Anyway, it indicates that the version on the CF Downloads page is updated, and as you may know that page has things RELATED to CF (like the lockdown tool) but not CF installers themselves. And it has sections for each of CF2023 and 2021.

3a) All that said, another unfortunate challenge for you is that they never put in the name of the file any indication of "what level of update" the file is.  Some may think that the file metadata (such as via properties>details in Windows) would tell you, but I often find that "version" value never changes even when I KNOW the file has changed.

Of course, you can try to guess from the size of the file (comparing one to another), or perhaps the checksum values (if Adobe offers it for a given file, and you know how to calculate yours. I have a post on that.)

Some might know that installers are zips (Windows exe's are, at least), and you can explore those to try to make out dates of file witin them--though they are a bit of a rats nest of folders and nested zip files.

4) As for the autolockdown tool, you would not be the first to report suprising (even sometimes devestating) problems with it. I regard it as something to approach with care, and indeed I always recommend people make a backup (or snapshot) before applying it. And no, it's not enough to just backup the CF folder: the tool touches much more than that, about the web server, the OS, the file system.

But to your specific issue, if you at LEAST saved off a copy of the neo-datasource.xml file (once you confirmed the DSNs were ok--preferably after a restart following the CAR import, just to be sure), then at least when things went south, you could try putting that back in place and restarting CF. It would be interesting if you saved off the "bad" one first, to compare them and see what's different.

And yes, I do expect the difference could be there, rather than in CF's sql server "package" itself (especially if you remove and re-add it and things still fail).

Anyway, those are a few thoughts for you. I do look forward to hearing if you resolve the matter one way or another.

[PatrickHolway]

Thanks for the responses. So a couple things so it's clear.

The data sources are always both visible in the CF adminUI and in neo-datasource.xml when I open it. Before and after I run the lockdown. Beofe i run the lockdown I can click validate in the cf admin UI and the data sources connect, after the lockdown runs, clicking that throws an error. 

I did confirm I am using the latest version of the lockdown installer.

The only thing that fixes it is re-installing the sqlserver package. Its really weird.

More to come

[Charlie Arehart]

OK on all that, and it may well really be down to the autolockdown tool, since you feel that simply reinstalling the sqlserver agent solves things. The CAR import wouldn't affect that at all. (Still, a restart before the lockdown might not be a bad idea just to PROVE that the DSNs still verify beforehand.)

FWIW, your subject did say the "datasourcs" were "nuked". It's now more clear that's not really the issue, right? Just the ability to verify (or add) them, which seems to be about the sqlserver package being nuked. 🙂

One last thing: the processing of packages happens during CF startup. You may want to look at your coldfusion-out.log and/or coldfusion-error.log after the autolockdown tool restarts CF. Perhaps you may see something during that startup that suggests more specifically what's amiss--though it may not tell you WHY.

As always, just tying to help. Got my popcorn ready for the next episode. 🙂

[BKBK]

@PatrickHolway , hats off to you for an exemplary description of a software issue. The readability and conciseness will help a lot of fellow ColdFusion Developers.

[PatrickHolway]

thanks @BKBK a few weeks ago I knew nothing about adminstering CF and now I know next to nothing 🙂

Ok so my latest adventure. New Tasks Server. This is a pretty basic CF install with a single IIS site that runs cron like jobs.

I'll cut to the chase here as it is a slightly different scenario.

1) Coldfusion 2021 Update 12 Installed and running with open jdk

2) Time to import the car file from the current prod tasks server

3) car file imported. WITHOUT restarting CF I check the data sources , they look good (is it a mirage?) -- screenshot 1 labeled DataSources-After-Car-Import-New-Install.png  

4) Since @Charlie Arehart suggested it, I restarted CF after the car file import and checked the Data sources. This time they dont validate. Screenshot labeled after-RESTART-of-CF.png 

5) well well well, mabybe it wasnt the lockdown installer but the restart after teh car import ??? 

6) Lets reinstall the sql sever package and restart cf and see what happens. screenshot labeled reinstall-sql.png - resintalls fine.

7) Restart CF and check datasources - and they work. screenshot labeled working.png 🙂 

8) check my healthcheck and pdf generator - yep they work.

So it seems after I import a car the datasources validate but they dont after a restart. I did not check the xml file sadly before and after the restart , i will next as I have more servers to build. This product sure does keep you on your toes. Enjoy your weekends !

[Charlie Arehart]

How about that? Glad I anticipated that for you. Let's see what more you learn.

(I hope people seeing the previous reply thread where we discussed those things will think to jump down to this other thread where you're replying to bkbk pretty separately. Yep, even these forums can "keep you on your toes".)