Coldfusion 2018 Admin page give 500 error after hotfix 16

[AJIT5CDC]

HTTP Status 500 – Internal Server Error

---

Type Exception Report

Message coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Runtime service is not available.

Description The server encountered an unexpected condition that prevented it from fulfilling the request.

[AJIT5CDC]

Exception

javax.servlet.ServletException: coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Runtime service is not available.
	coldfusion.bootstrap.ClassloaderHelper.initServletClass(ClassloaderHelper.java:129)
	coldfusion.bootstrap.BootstrapServlet.init(BootstrapServlet.java:111)
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:377)
	org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:463)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)

Root Cause

coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Runtime service is not available.
	coldfusion.server.ServiceFactory.getRuntimeService(ServiceFactory.java:121)
	coldfusion.CfmServlet.init(CfmServlet.java:101)
	coldfusion.bootstrap.ClassloaderHelper.initServletClass(ClassloaderHelper.java:121)
	coldfusion.bootstrap.BootstrapServlet.init(BootstrapServlet.java:111)
	org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:541)
	org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
	org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:377)
	org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:463)
	org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
	org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:889)
	org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1743)
	org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
	org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
	org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
	org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	java.lang.Thread.run(Thread.java:748)

[AJIT5CDC]

Aug 14, 2023 3:51:18 PM org.apache.catalina.core.ApplicationContext log
INFO: failed to load: coldfusion.CfmServlet
Aug 14, 2023 3:51:18 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Allocate exception for servlet [CfmServlet]
coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Runtime service is not available.

[RaviShankar Chagnur]

Hello Ajit,

Can you please start ColdFusion instance through command prompt using cfstart.bat command and share the output

Thanks & Regards

Ravi

[AJIT5CDC]

# ./cfstart.sh

Starting ColdFusion 2018 server ...

The ColdFusion 2018 server is starting up and will be available shortly.

======================================================================

ColdFusion 2018 server has been started.

ColdFusion 2018 will write logs to /opt/coldfusion2018/cfusion/bin/../logs/coldfusion-out.log

======================================================================

Following is out put of /opt/coldfusion2018/cfusion/bin/../logs/coldfusion-out.log

08/14 16:07:51 INFO Starting Flex 1.5 CF Edition

Aug 14, 2023 16:07:51 PM Information [main] - Starting logging...

Aug 14, 2023 16:07:51 PM Information [main] - Starting license...

Aug 14, 2023 16:07:51 PM Information [main] - Enterprise Edition enabled

Aug 14, 2023 16:07:51 PM Information [main] - Starting crypto...

Aug 14, 2023 16:07:52 PM Information [main] - Installed JSafe JCE provider: Version 6.21 Crypto-J 6.2.1, EMC Corporation. JsafeJCE Security Provider (implements RSA, DSA, ECDSA, Diffie-Hellman, ECDH, AES, DES, Triple DES, DESX, RC2, RC4, RC5, PBE, MD2, MD5, RIPEMD160, SHA1, SHA224, SHA256, SHA384, SHA512, HMAC-MD5, HMAC-RIPEMD160, HMAC-SHA1, HMAC-SHA224, HMAC-SHA256, HMAC-SHA384, HMAC-SHA512, HMACDRBG, HASHDRBG, CTRDRBG, FIPS186PRNG, SHA1PRNG, MD5PRNG; RFC 3394, RFC 5649 AES Key Wrap; X.509 CertificateFactory; PKCS12, PKCS15 KeyStore; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathValidators; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathBuilders; LDAP, Collection CertStores)

Aug 14, 2023 16:07:52 PM Information [main] - Starting security...

Aug 14, 2023 16:07:52 PM Error [main] - Unable to initialise Security service: java.lang.NoSuchFieldError: TRACE

Aug 14, 2023 16:07:52 PM Information [main] - Unable to initialise CFStartupServlet:TRACE

Aug 14, 2023 16:07:52 PM Information [main] - ColdFusion: application services are now available

08/14 16:07:52 INFO Macromedia Flex Build: 87315.134646

[RaviShankar Chagnur]

Hello Ajit,

Can you please check if there are any duplicate entries for Log4j jar files in cfusion/lib directory.

Please share output of "ls -lrt" command for cfusion/lib directory

Thanks 

Ravi

[Charlie Arehart]

If Ravi's suggestions may not get you to resolution, I'd propose instead you focus on the assertion that this happened with update 16. First, what update had you been on before that?

Second and more important, what is reported in the log from that update. To be clear, you would look at opt/coldfusion2018/cfusion/hf-updates and its folder for update 16. In that will be an install log, and in that will be a count of fatalerrors and nonfatalerrors. Those should both be 0. If they are not, then there were errors during the install of the update. There can be many causes for that,.and solutions. Let's hear what you find, first. 

[AJIT5CDC]

Updating from HF 10 to HF 16. I successfully did on several servers but some of the servers same operation is failing.

Here is the output of install log.

Mon Aug 14 03:24:35 EDT 2023

Free Memory: 18951 kB
Total Memory: 61440 kB

java.class.path:
/opt/coldfusion2018/cfusion/hf-updates/hotfix-016-330130.jar

ZGUtil.CLASS_PATH:
/opt/coldfusion2018/cfusion/hf-updates/hotfix-016-330130.jar

sun.boot.class.path:
<none specified>

java.ext.dirs:
<none specified>

java.version == 11.0.1 (Java 1)
java.vm.name == Java HotSpot(TM) 64-Bit Server VM
java.vm.vendor == Oracle Corporation
java.vm.version == 11.0.1+13-LTS
java.vm.specification.name == Java Virtual Machine Specification
java.vm.specification.vendor == Oracle Corporation
java.vm.specification.version == 11
java.specification.name == Java Platform API Specification
java.specification.vendor == Oracle Corporation
java.specification.version == 11
java.vendor == Oracle Corporation
java.vendor.url == http://java.oracle.com/
java.class.version == 55.0
java.library.path == /usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
java.compiler == null
java.home == /opt/coldfusion2018/jre
java.io.tmpdir == /tmp
os.name == Linux
os.arch == amd64
os.version == 3.10.0-1160.92.1.el7.x86_64
path.separator == :
file.separator == /
file.encoding == UTF-8
user.name == root
user.home == /root
user.dir == /opt/coldfusion2018/cfusion/logs
user.language == en
user.region == null
__________________________________________________________________________

Installed Feature(s) Standard, J2EE of Adobe ColdFusion 2018 Update 16

Install Begin: AUGUST 14, 2023 3:23:02 PM EDT
Install End: AUGUST 14, 2023 3:24:32 PM EDT

Installed by InstallAnywhere 20.0 Premier Build 6376

User Interactions
-----------------

#Choose ColdFusion 2018 Home directory
#--------------------------------------
USER_INSTALL_DIR=/opt/coldfusion2018

#Install
#-------
-fileOverwrite_/opt/coldfusion2018/notice.txt=Yes

Summary
-------

Installation: Successful.

3090 Successes
0 Warnings
0 NonFatalErrors
0 FatalErrors

Action Notes:

None

[Charlie Arehart]

Interesting. So that confirms that yes, there were 0 fatalerrors or nonfatalerrors during the update. But I see it also indicating that you're running Java 11.0.1. What about the other machines that "worked" for you? Maybe they are on more recent updates to Java 11. (The latest, from last month, is 11.0.20.) 

As for that 11.0.1 used here, that's the original jvm that came with your cf2018 installer, and it's 5 years old. If you read the Adobe security bulletins (pointed to by many of the technotes for the updates you're applying), they indicate how just updating cf is not enough and that you should also be keeping that Java 11 updated.

I discuss that process and point to resources with still more at https://www.carehart.org/cfupdate/#java . I also offer direct help there, if you "just want to get this done, without a lot of homework". 🙂 

[AJIT5CDC]

Java version is same for both working and not working servers. 

[Priyank Shrivastava.]

Hi @AJIT5CDC ,

It seems like you are going from HF10 to HF16 which may have caused this issue. Could you please uninstall this update and install update 12 and then move to update 15 and then later move to update 17.

[Charlie Arehart]

Separate from Priyank's subsequent suggestion, I'd want to confirm first how you're judging that the Java is the "same". I realize you know the same Java is on both machines.

But can you look at what's reported in the cf admin "settings summary" page? Cf could be pointing to a JVM other than what you expect. Second, can you look in the same update log on the working server? The jvm used by that could be OTHER than what's used by cf, as you or someone could have applied the update from the command line. That could cause it to use a different version than the 11.0.1 reported above.

Just offering this for the sake of completeness. 

[ccgmaster]

I have a similar question regarding manual updates.  I must apply manual update 7 on 2018 due to the UI issue on the hotfix screen in CF2018 admin.  The instructions state to use the JVM that came installed with CF, but the JVM actually points to a different location.  should the newer JVM be used to manually install the update, or the stock on that came with CF?   long story short, also need to install update 16 but cannot get to it through the UI.  attached is a summary of what I'm trying to accomplish