Coldfusion 2021 slow startup after migration from 2018

[C.Barrow]

We did an upgrade last night from Coldfusion 2018 to CF 2021.  The installation upgrade went fine but now it takes several minutes, 3 to 5 minutes, for the application server to start and sometimes we have to restart it because it fails.  The CF administrator is working as well as the website.  There is no noticeable performance differences in that part once the cf 2021 app server was running, however, when we tried to start appling the service releases, we only made it up to release 5 before it failed to start back up.  Each patch release said it completed without issue but it would just take a while for the app server to reload.  The Virtual server has dual CPUs allocated, 8GB of memory, and 30GB of C drive storage space. Does anyone have any suggestions of where to look to find the issue?

[Charlie Arehart · Accepted solution]

Is the cf2021 running in the same server as cf2018 was? If not, you may find that cf2018 would ALSO start slow. It may be worth proving, so you don't focus needlessly on the issue being cf2021 itself.

Either way, as for what can cause the slowness, there are MANY potential reasons, too many to list. My bet would be disk activity, or network activity within or to outside your network.

The cf logs will tell you where time is spent on each part of the startup process. Where do they show a gap?

As for problems after update 5, there's nothing about it to contribute to this. BTW, you could have just done the latest. Anyway, there is an install log for each update applied, within the hf-updates folder. Each should report 0 fatalerrors. Do they?

Finally, if you "just want this solved" without a lot of back and forth, I can help directly via remote consulting. More on rates, approach, satisfaction guarantee, online calendar, and more at carehart.org/consulting. I can even meet this weekend if it's urgent for you. 

[C.Barrow]

Thanks Charlie for your help and insights on this.  I did this as an inplace upgrade on the same Virtual server and to be honest, if there was a lag before, I was unaware.  I can say that I didn't see it when I was doing the service releases updates on CF 2018.  I also agree that service release 5 shouldn't have caused it to stop trying to start back.  I also looked through those update logs and didn't see any errors.  I have gone through some of the log files and did notice a

couple things. 

   First, I noticed that it's still trying to activate my license.  Our server can get to the internet but for some reason, the Coldfusion Administrator can't see the adobe site and can't activate the new serial number.   It was spending a lot of time trying to do the activation and eventually times out.  This was the same under CF 2018 as well.  I always went out and just downloaded the jar file then did a manual update.  Is there something I need to do in CF Admin or somewhere else to get this part resolved?  We are still running on the eval trial, but I had put that on the back burner for the moment trying to figure out the other issues first.

   Secondly, we are using IIS as our web server and not the native Apache server,  however  I did find the following error listed in the coldfusion-error log

SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@37173772]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@5994a67c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

This was also happening during the start up.

I didn't know if either of these two would cause the slowness issues or what would be the cause for the second issue.

[Charlie Arehart]

A few things. 

1.  Ok that you don't recall the same slowness with cf2018. (My first paragraph asking about that was indeed just for that question. The rest was regardless of the answer.)
2.  As for your looking in the logs and finding "no errors", please look again--and you need only look at the latest update install logs for whatever latest update cf2021 is on. Again, look specifically "0 fatalerrors". It's a table at about line 70 of the update install log.
3.  As for your observation of slowness in activation of the licensing, no that did NOT happen with CF2018. Activation is new with cf2021. (What you recall about copying update files is a totally separate matter, which could affect both). And I'd argue this activation matter could be YOUR ENTIRE ISSUE with slowness to startup since it would be new to you with cf2021. More on that, here. 
4. As I said, look at the cf logs during startup. That will show where gaps in time are. Focus specifically on the coldfusion-error.log and the coldfusion-out.log, during the last cf startup. Please don't skip this step.
5. Then yes, there is also a license.log that you should check, for activation matters. What does IT  show during startup?  And I said not to skip the last step because your slowness may NOT be due to activation. Let's confirm.
6. Finally, about the last error you reported, you can ignore that. Look closely: that's during shutdown of cf, not startup. Yes, it looks scary. Yes, it refers to memory leaks. No, it's NEVER been useful or meaningful to me in the 12 years that cf has been running on tomcat (where this is reported because a certain "valve" is enables to report such "leaks" during shutdown). Please just ignore it, for this problem.

Let us know how things go/what you find. 

[C.Barrow]

Just a quick note on this discussion.  The  coldfusion-error.log and the coldfusion-out.log files both pointed the issue to being with the application trying to reach out to adobe for activation.  After I was able to get the serial number validated and activated, the application startup was much quicker and more like normal.

[Charlie Arehart · Accepted solution]

Thanks for the update, and good to see what you found confirmed what I proposed (in step 4 of my previous reply. It may have been interesting to hear what you found in step 5 also, the activation.log).

Anyway, I see someone marked my first reply as "the answer", but actually I think the second one would be better, given what you have confirmed here. If that was you, and you see my reply here and agree, I hope you'll consider changing it. You could even mark your last reply and my previous comment as both being "the answers", since mine proposed the solution and yours confirmed it, then unmark my first reply as being the answer. Your call. This comment may suffice for some readers.