Installed Coldfusion 2018 Update 14 an now get Monitoring Service is not available error

[mwoods1971]

Server: Linux

Codfusion 2018 Update 14 installed from the command line.

The update said it completed succesfuly and when tryign to get to the admin page, I get an error 

that says: The Monitoring Service is not available.

Please note: CF is service pages, but I just can't get to the admin. A dump of the server variable confirms that update 14 is installed.

Logs:

May 13, 2022 18:25:20 PM Information [main] - Starting logging...
May 13, 2022 18:25:20 PM Information [main] - Starting license...
May 13, 2022 18:25:20 PM Information [main] - Enterprise Edition enabled
May 13, 2022 18:25:20 PM Information [main] - Starting crypto...
May 13, 2022 18:25:20 PM Information [main] - Installed JSafe JCE provider: Version 6.21 Crypto-J 6.2.1, EMC Corporation. JsafeJCE Security Provider (implements RSA, DSA, ECDSA, Diffie-Hellman, ECDH, AES, DES, Triple DES, DESX, RC2, RC4, RC5, PBE, MD2, MD5, RIPEMD160, SHA1, SHA224, SHA256, SHA384, SHA512, HMAC-MD5, HMAC-RIPEMD160, HMAC-SHA1, HMAC-SHA224, HMAC-SHA256, HMAC-SHA384, HMAC-SHA512, HMACDRBG, HASHDRBG, CTRDRBG, FIPS186PRNG, SHA1PRNG, MD5PRNG; RFC 3394, RFC 5649 AES Key Wrap; X.509 CertificateFactory; PKCS12, PKCS15 KeyStore; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathValidators; X.509V1, PKIX, PKIX-SuiteB, PKIX-SuiteBTLS CertPathBuilders; LDAP, Collection CertStores)
May 13, 2022 18:25:20 PM Information [main] - Starting security...
May 13, 2022 18:25:20 PM Information [main] - Starting scheduler...
May 13, 2022 18:25:20 PM Information [main] - Starting WatchService...
May 13, 2022 18:25:20 PM Information [main] - Starting debugging...
May 13, 2022 18:25:20 PM Information [main] - Starting sql...
May 13, 2022 18:25:21 PM Information [main] - Starting runtime...
May 13, 2022 18:25:22 PM Information [main] - CORBA Configuration not enabled
May 13, 2022 18:25:22 PM Information [main] - Starting mail...
May 13, 2022 18:25:22 PM Information [main] - Starting cron...
May 13, 2022 18:25:23 PM Information [main] - Created scheduler DefaultQuartzScheduler with thread pool size as 10
May 13, 2022 18:25:23 PM Information [main] - Starting registry...
May 13, 2022 18:25:23 PM Information [main] - Starting client...
May 13, 2022 18:25:23 PM Information [main] - Starting xmlrpc...
May 13, 2022 18:25:23 PM Information [main] - Starting jaxrs...
May 13, 2022 18:25:23 PM Information [main] - Starting graphing...
May 13, 2022 18:25:23 PM Information [main] - Starting solr...
May 13, 2022 18:25:23 PM Information [main] - Starting archive...
May 13, 2022 18:25:23 PM Information [main] - Starting document...
May 13, 2022 18:25:26 PM Information [main] - Starting eventgateway...
May 13, 2022 18:25:26 PM Information [main] - Event Gateway Disabled.
May 13, 2022 18:25:26 PM Information [main] - Starting FlexAssembler...
May 13, 2022 18:25:26 PM Information [main] - Starting .NET...
May 13, 2022 18:25:26 PM Information [main] - Starting Monitoring...
May 13, 2022 18:25:27 PM Error [main] - Unable to initialise Monitoring service: java.lang.NoSuchMethodError: org.apache.log4j.helpers.OptionConverter.convertLevel(Ljava/lang/String;Lorg/apache/logging/log4j/Level;)Lorg/apache/logging/log4j/Level;
May 13, 2022 18:25:27 PM Information [main] - Starting PDFG...
May 13, 2022 18:25:27 PM Information [main] - Starting WebSocket...
May 13, 2022 18:25:28 PM Information [main] - WebSocket server listens on port: 8581
May 13, 2022 18:25:28 PM Information [main] - ColdFusion started
May 13, 2022 18:25:28 PM Information [main] - ColdFusion: application services are now available
05/13 18:25:28 INFO Macromedia Flex Build: 87315.134646
May 13, 2022 18:25:31 PM Error [Thread-15] - Registration error for service manager : .http://127.0.0.1:8991/PDFgServlet/.Reason: NOT FOUND
May 13, 2022 18:26:28 PM Error [http-nio-8500-exec-1] - The Monitoring service is not available.This exception is usually caused by service startup failure. Check your server configuration. The specific sequence of files included or processed is: /data/coldfusion2018/cfusion/wwwroot/CFIDE/administrator/index.cfm, line: 114
May 13, 2022 18:27:55 PM Error [http-nio-8500-exec-4] - The Monitoring service is not available.This exception is usually caused by service startup failure. Check your server configuration. The specific sequence of files included or processed is: /data/coldfusion2018/cfusion/wwwroot/CFIDE/administrator/index.cfm, line: 114
May 13, 2022 18:36:23 PM Error [http-nio-8500-exec-6] - The Monitoring service is not available.This exception is usually caused by service startup failure. Check your server configuration. The specific sequence of files included or processed is: /data/coldfusion2018/cfusion/wwwroot/CFIDE/administrator/index.cfm, line: 114

May 13, 2022 18:55:22 PM Error [http-nio-8500-exec-8] - The Monitoring service is not available.This exception is usually caused by service startup failure. Check your server configuration. The specific sequence of files included or processed is: /data/coldfusion2018/cfusion/wwwroot/CFIDE/administrator/index.cfm, line: 114
May 13, 2022 19:03:56 PM Error [Thread-19] - The Monitoring service is not available.
May 13, 2022 19:03:56 PM Information [Thread-19] - ColdFusion stopped

I have tried restarting the service multiple times but still getting the same result. 

[Dave Watts]

I'm a little over my head here, but I think you might want to reach out to CF support or open a ticket at https://tracker.adobe.com/. My read of your error is that replacing the log4j version to fix the recent vulnerability may have broken something in the monitoring service. Of course, you'd think that wouldn't have made it out of Adobe's testing if it affected everybody, and my read of your error could be entirely wrong. So if you do open a ticket, just tell them exactly what you're seeing, not what I'm thinking is happening.

Dave Watts, Eidolon LLC

[mwoods1971]

Hey Dave,

Thanks, I just sent that off to cfsup@adobe.com  with the detials listed here.

Thanks for the suggestion! 

[Dave Watts]

Good luck! Make a note here when you get a resolution or have followup questions.

Dave Watts, Eidolon LLC

[Charlie Arehart]

Fwiw, that error is not at all unusual, and it can have various causes.

And while Dave makes a reasonable assertion about the reference to log4j and the update being related to that, I'll just note that I've seen that error about monitoring and log4j also in other threads here recent years past, with even cf2016 and 11 (so again, not new). 

So what might it be instead? Well first, you should check if there may have been an error during the applying of that update. They happen often, and it's not unusual for an odd error like you're reporting to have such an error during the update to be the root cause. And I have a blog post on finding and dealing with such update errors (which log to look at, which is not obvious, as well as where in the log to look, also not obvious, as well as some proposed workarounds for common update problems) :

How to solve common problems with applying ColdFusion updates 

Let us know what you find and how it goes, or if you learn of something else. 

[mwoods1971]

@Charlie Arehart @Dave Watts 

I was able to find the log for the update and there are two no-fatal errors. The log is attached but here are the two fails:

Execute ANT Script: Script: unix_basic_commands.xml
Status: ERROR
Additional Notes: ERROR - unix_basic_commands.xml:222: Execute failed: java.io.IOException: Cannot run program "/tmp/184194.tmp/temp-init.sh" (in directory "/tmp/184194.tmp"): error=13, Permission denied

ANT Script Error:
Status: ERROR
Additional Notes: ERROR - unix_basic_commands.xmlExecute failed: java.io.IOException: Cannot run program "/tmp/184194.tmp/temp-init.sh" (in directory "/tmp/458590.tmp"): error=13, Permission denied

[BKBK]

There's something not quite right. You presumably installed Update 14 of CF2018 in May 2022. However, the installation log that you have attached is dated November 14, 2020.

I would suggest you uninstall the update then re-install it following the instructions Adobe has provided.

[mwoods1971]

So, that's probably becasue this instance was moved over from another machine. We upgraded from one OS to another. However, I'm more thank willing to try anything. I'll read the link. Thanks

[Yodeler]

I am having the exact same problem.  After getting the error, I tried making sure the server was stopped (using cfstop) then uninstall, then reinstalling, restarting the services and I still have the problem.  The server is running is just gets the error that the monitoring service is unavailable and the administrator is not accessible.  Since the error is related to log4j, it might have something to do with the fact that we replaced the log4j file with the downloaded one from Adobe because the one that came with coldfusion after update 13 was not acceptable to our scanner (no exceptions).  It ran fine until this update.  Any ideas?

[Yodeler]

BTW - all the install log files look clean.

[BKBK]

@mwoods1971 , ColdFusion may be able to serve pages after the update, but I would consider this installation to be unreliable. That is because it has suffered a startup failure.   

Your logs imply it is caused by a bug. Take, for example, "/data/coldfusion2018/cfusion/wwwroot/CFIDE/administrator/index.cfm". This path points to the Administrator. Yet the installation refers to it as a configuration file. Which I think is nonsensical.

I agree with Dave that this bug - which it is in all likelihood - is related to changes in the Log4J library. That, combined with errors during the loading or processing of the monitoring file, neo-monitoring.xml.

So, @mwoods1971 , like Dave, I too would suggest you open a bug ticket. Remember to copy-paste the logs in the ticket. It might also help to include in it a link to this forum discussion.

Oh, and please share the link to the ticket afterwards.

[Yodeler]

I submitted a bug for my version of it.  https://tracker.adobe.com/#/view/CF-4213692

[Yodeler]

If you look at the log files, there is also this error in the scheduler log:  "log4j-slf4j-impl cannot be present with log4j-to-slf4j"

[Charlie Arehart]

Yodeler (and others), I think you've identified the problem. I've seen this more and more as people scramble to override what Adobe has done, intentionally or not.

You say, "it might have something to do with the fact that we replaced the log4j file with the downloaded one from Adobe because the one that came with coldfusion after update 13 was not acceptable to our scanner (no exceptions)."

Yep. So let's clarify: you're now doing update 14, right, like mwoods was? That update should remove/replace whatever log4j files were put in place by update 13...or by your putting them in place per the two technotes after that for log4j. But if you or anyone did anything regarding those log4j files beyond that, the updater won't know and would leave those in place. And you could have classloading conflicts of the sort indicated in the error messages above. 

In particular, here's what may be tripping some people up: if you rename old jar files, leaving them in place, java could still load them. First, it doesn't care (at all) about filenames. Second, I've seen even it not care about file extensions either, still loading a class unexpectedly from a file called something.jar.old.

So I'd recommend anyone facing this problem carefully assess what log*. * files exist in the cfusion/lib (or [instance name] /lib). Remove any that seem questionable and restart cf, to see if things now work. Don't delete them, move them--in case you don't solve this problem (or later find some other) and you need to put them back.

Even better would be to compare your lib folder to one created and updated without touching it otherwise. That may mean installing the free trial/Dec edition of cf somewhere else, sure. Or those familiar with containers /docker images can use for comparison the file system within the Adobe-provided images, which already have the updates in place.

Hope that may be enough to get folks going, or I can help folks who may "just need to get this solved" without all the back and forth, via carehart.org/consulting. Or perhaps Adobe or others may offer to help for free. 

[Yodeler]

Charlie, thanks so much.  Someone else had installed the downloaded log4j file but left it in the lib folder.  After I moved it, it works.

[Charlie Arehart]

There you go. Glad to have helped, and thanks for the confirmation. 

[BKBK]

Hi @Yodeler , thanks for the link to the bug ticket. Out of it comes a suggestion for @mwoods1971 

@mwoods1971 , if I were you I would have a look at all the logs, one by one. You might find an error message as decisive as the one Yodeler found.

[Charlie Arehart]

FWIW, the log message in mwoods' first message here clarifies it being a log4j issue. And since yodeler said he had "the exact same problem",  I'm hopeful that mwoods will find their solution to be the same as what I suggested above to yodeler (who confirmed a

that it led to his resolution).

So I'd propose that simple suggestion be considered before any further digging into logs or more, by him or others here.

Again, this problem is becoming common as people scramble (or did in the past scramble) to address log4j problems, sometimes creating the very holes they then later fall into.