ColdFusion service will not start

[coolidge]

Hello - We're running ColdFusion Standard 2018.  Early yesterday morning the service stopped, and we have not been able to start it.  Is anyone familiar with these errors?  Thank you.

Apr 30, 2023 2:02:28 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Apr 30, 2023 2:02:28 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-8500"]
Apr 30, 2023 2:02:28 AM org.apache.coyote.AbstractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-nio-127.0.0.1-8018"]
Apr 30, 2023 2:02:28 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Catalina]
Apr 30, 2023 2:02:34 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@480d45d0]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@27382402]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 30, 2023 2:02:34 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@480d45d0]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@2fb1a241]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 30, 2023 2:02:35 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@480d45d0]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@4cfcd17d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 30, 2023 2:02:35 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@480d45d0]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@43fbccef]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Apr 30, 2023 2:02:35 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8500"]
Apr 30, 2023 2:02:35 AM org.apache.coyote.AbstractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-nio-127.0.0.1-8018"]
Apr 30, 2023 2:02:35 AM org.apache.catalina.loader.WebappClassLoaderBase checkStateForResourceLoading
INFO: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at coldfusion.bootstrap.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:218)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at coldfusion.security.SecurityManager$4.run(SecurityManager.java:1849)

Apr 30, 2023 2:02:35 AM org.apache.catalina.loader.WebappClassLoaderBase checkStateForResourceLoading
INFO: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
java.lang.IllegalStateException: Illegal access: this web application instance has been stopped already. Could not load [java.util.concurrent.ScheduledFuture]. The following stack trace is thrown for debugging purposes as well as to attempt to terminate the thread which caused the illegal access.
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForResourceLoading(WebappClassLoaderBase.java:1427)
at org.apache.catalina.loader.WebappClassLoaderBase.checkStateForClassLoading(WebappClassLoaderBase.java:1415)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1254)
at org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1215)
at coldfusion.bootstrap.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:218)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at coldfusion.security.SecurityManager$2.run(SecurityManager.java:1757)

[coolidge]

I should also mention it appears the latest Microsoft security patches have caused this issue.  Since we applied the patch we have not been able to re-start the service.  We can roll back the patch of course, but if anyone knows a way to resolve this that would be preferrable.  Thank you.

[Charlie Arehart]

Those errors are from the shutdown, and nearly all are very common. I suggest you focus solely on why it won't come up. 

And for that, the first thing I'd suggest is go to the command line (as admin) and to cf's cfusion/bin folder. There, run cfstart. What does THAT report?

My money is that the problem is about the jvm/java engine or jvm args that CF is using. There can be many possible explanations (even with that) for things "suddenly" failing. I would NOT  think it had anything to do with any ms security update. But if the box rebooted after that, this is a problem that could have been caused days or weeks ago, if cf (and/ the box) had not been restarted since then--and this first restart since then is simply tripping over what was done then. Again, we can't know but perhaps that will help you assess more than just "what was the last change we made" (a common mistake in this situation).

Finally, I'd put my chips directly on this: someone installed a new Java version, running the jdk installer for Java 11.0.19 or 18. And cf is pointing to a folder whose name is for an older jvm version which is either gone or now nearly empty, such as jdk-11.0.17 or earlier. If there is now a sibling jdk-11 folder, point cf to that and try starting it (as a service, rather than from the command line).

If that does it, see a blog post I did in January for more on this jvm change (and how it would affect cf folks like this).

https://www.carehart.org/blog/2023/1/23/beware_latest_oracle_JDK_installers_will_remove_older_JDK_in...

But if that's not it, let us know what the cfstart at the cmd line shows, if anything. Hope something I've share may help you or future readers. 

[paule12345]

Thanks for the cfstart tip, Charlie.  Ran windows updates on Windows 2019 server today, restarted the machine and the CF 2021 (Update 10) service would not restart.  Nothing written to the coldfusion-error.log.  Ran cfstart from the command line and it mentioned an access violation error in the java runtime environment.    Oddly enough a reinstall of JDK 11.0.20 fixed the problem.   

[Charlie Arehart]

Thanks for confirming the command line cfstart helped you, Paul. 

FWIW, as with Coolidge in May, I have my doubts it was the windows update itself, but instead that something else changed before the box (and therefore CF) restart. But I appreciate you may be happy enough just to be running again, without really wanting to dig into "what really happened". 🙂 Glad to hear that you're back up and running. 

[BKBK]

The stacktrace seems to suggest that one of the threads was still hanging when you last stopped or restarted ColdFusion. I would suggest that you restart Windows. That should stop any hanging threads.

[coolidge]

Thank you both for your help.  We tried all suggestions here and nothing worked.  So we re-installed ColdFusion Server 2018.  The service starts now which is the good news.  The bad news is that ColdFuion files do not run (see below).  All CF Web app files are in the same location as before the installation.  We can run admin files through port 8500, but the regular cf app files do not run.  I haven't installed CF in so long - I feel like we're missing something obvious.  What do we need to do to simply run CF files?

404 - File or directory not found.

The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.

[Charlie Arehart]

I appreciate how frustrating this is. It all can be solved. I help people do it daily.

First, FWIW you never said what you found when you ran the cfstart  command I'd suggested, or what jvm cf was pointing to (details via info or links in the blog post). Perhaps you were in a hurry and "didn't have time" to read it. And I realize it's too late now. 

As for your current situation, we can only guess but I suspect it's that you need to run the cf web server configuration (wsconfig) tool. That's not done automatically in the installer since cf2016. Are you familiar with it? There are docs and other resources on using it, including from myself, or Adobe, or others. That may be all you need to do. 

That said, I fear you probably didn't think to run the tool BEFORE reinstalling cf, so that it would UNINSTALL your previous working web server configuration, and now you may find that trying to run it won't work because it trips over the old config in iis, pointing perhaps to wsconfig folders in cf that were removed in your uninstall of cf. 

Walking through all you need to do to recover (if this happens) is too much to anticipate and write. But let's see if just having you run the wsconfig tool is enough to get you going. 

Finally, if you "just need it working" versus lots of back and forth here over hours or days, I can help directly, remotely, and quickly, with satisfaction guaranteed (you won't pay for time you don't find valuable). More at carehart.org/consulting, and if you want to work tonight rather than choose from my online calendar, drop me an email via the email address offered there also. 

[BKBK]

Re-installation? Wouldn't that have undone all previous updates?

As you are probably aware, installing ColdFusion 2018 is far from being enough. Have you installed all the CF 2018 updates and security patches?