ColdFusion 2018 update 14 500 errors

[Todd in Italy]

I have a default install of Coldfusion 2018 update 13 running without issues. After the update 14 is installed manually via the jre/bin/java.exe, both the admin page and the coldfusion site throw 500 errors. I tried increasing the debug info from the admin panel before beginning, but it still just shows 500. I also made sure my c++ runtime is up to date. I also tried rebuilding the connector to IIS, even though I should not have to as the one from update 13 should work. Nothing fixes the 500 error, though uninstalling update 14 restores functionality. Any other ideas how get this update to work, or where to look to find out more details about what is actually broken?

[Dave Watts]

My initial thought is that you have cached content from the old version, and the new version can't run it. So, you might try clearing your cache before updating.

Dave Watts, Eidolon LLC 

[Todd in Italy]

Thank you, but that is not the issue. The 500 error is persistent across multiple browsers regardless of the cache. 

[Dave Watts]

The "caching" I'm talking about is server-side compilation to Java classes, which is different from browser caching. There's a caching page in CF Administrator under Server Settings, I think. And that should have an option to clear the server cache.

Dave Watts, Eidolon LLC 

[Todd in Italy]

I see. Thanks for clarifying. Unfortunately, once the update has run, the admin page also gets a 500 error, so there is no way to access the server cache clearing function. I can try clearning it before running the update, or after if someone can tell me how to do so manually.

[Todd in Italy]

Clearing the server side cache before the update did not make a difference. 

[Charlie Arehart]

As for the possibility of an error in applying the update, look to the update log, in the cf hf-updates folder, which has a subfolder for each update you run. You will see an install log (with a long name) for each install and uninstall you've done. And about 70 lines down in the log file will be a count of successes, warnings, nonfatal, and fatal errors. You want 0 of the latter, while 1 for warning or nonfatal is probably nothing to worry about.

I'm assuming you ran the command line "as admin" where you ran the Java command, right? You do need to. That alone could be the problem.

Finally, since you get 500 errors in browsing pages (and assuming it's not cached content, if that's what Dave wonders), look to the cf logs. Don't trust the date/time modified. Check at least the application.log, coldfusion-out.log, coldfusion-error.log, server.log, and exception.log.

Let us know what you think or find. 

[Todd in Italy]

Thank you for the response.

Yes indeed, the CMD shell was run as admin. As far as I know, java.exe will not install a jar file without admin privs. In any case, the install ran and and exited with no errors shown.

The install log shows 3441 Successes, 0 warnings, 0 NonFatal and 0 FatalErrors

There are several errors showing in the cf logs, the most suspicious of which are in the exception.log and server.log at the time of failure,  "could not initialize class org.apache.log4j.LogManager"

Earlier in the year we manually cleaned up some log4j issues and may have damaged the default log4j installation. I had assumed that an update would rebuilt them as needed, but they may be attempting to modify something that is no longer there. Also, patch 13 was functioning perfectly when we started the update, so it seems odd that patch 14 would then fail.

Any idea how to repair the patch 13 install so that patch 14 will correctly install? Or is a complete reinstall called for now?

[Charlie Arehart]

Todd, bad news/good news.

First, I think what you've shared now tells us this is a problem of your own doing (someone on your end, even if not you yourself). The indication of the log4j errors is the first clue; the second is your acknowledgement that "we manually cleaned up some log4j issues".

The good news? No, I don't think you will need to do "a complete reinstall".

My first suggestion is to check if (in that manual remediation effort) someone renamed a log4j jar. Many tried that, which led to suxh problems (sometimes simply on a cf restart, other times after an update implemented a conflicting jar). Sadly, changing a jar's name doesn't keep Java from looking into it if it's still in a folder in the classpath.

CF has a few log4j files in its cfusion/lib folder, in particular. You'll likely just need to remove the one that's there which cf does not EXPECT to be there, and is now tripping over.

That may be literally all you need to do. I'm on my phone as I write, so can't offer the list of what log4j files SHOULD be there. Maybe someone else seeing this (or you, comparing a working one) can tell you, or I hope to later.

Finally, FWIW, you responded to one of my suggestions saying, "As far as I know, java.exe will not install a jar file without admin privs". First, running java -jar does not INSTALL a jar, it simply RUNS it. And you absolutely CAN run that from a cmd line when NOT "run as admin". That's easily confirmed. And my point was that the work DONE by the code in some jars (like a cf update jar) may need to perform actions that it can't if not "run as admin" (and assuming the user you're running as also can't).

Anyway, let us know if you might make progress now with what I've shared, or if you need that help of a list of the only log4j jars that SHOULD be there in CF. Hope this gets you going again. 

[Todd in Italy]

Hello, and thanks for the clear information. None of the jar files were renamed. As far as I know, none were added that should not be there. The only operation logged was the removal of some jar files, but the log is manually created, so it could be incorrect. If you or anyone else has a list of the log4j files that should be there, I can certainly check to see if there are any extraneous files. If not, I am leaning towards a full reinstall. Our installation is pretty simple, so it's just a matter or installing, setting up the database connection, and an IIS connector.

I understand what you are saying about exectuting a jar. I certainly do have occasion to execute jar files that run as regular users. However, this will never work when the jar file itself is an installation that requires admin rights to modify or write. So yes, I do understand why you asked the question, and it was reasonable to ask just in case a person did not know that. Your wording is far more precise than my own, so thanks for clarifying!

[BKBK]

Phew! Todd, I thought you had obtained 14500 errors after updating ColdFusion. 🙂

[Todd in Italy]

I wish! A complete failure to modify the files would be easier to troubleshoot than an install that reports "success" but won't run!

[BKBK]

I have a default install of Coldfusion 2018 update 13 running without issues. After the update 14 is installed manually via the jre/bin/java.exe, ...

By @Todd in Italy

Hmmm, dicey. Running Java to install ColdFusion raises some questions. For example, about rights. 

There is also the question of hotfixes. You would have had to re-instate them after moving from Update 13 to Update 14. For example, the 3 hotfixes on my CF2018 Update 13 were:

/cfusion/lib/updates/hf201800-4208163.jar

/cfusion/lib/updates/hf201800-4212383.jar

/cfusion/lib/updates/hf201800-4212487.jar

I backed up these JARs and reinstated them after installing Update 14.

Suggestion:

1.  If you have installed Update 14 and it isn't working: Open the ColdFusion Administrator. Use the Server Update page to uninstall Update 14. Restart ColdFusion. Verify that you're back to Update 13.
   Open the ColdFusion Administrator. Use the Server Update page to install Update 14.
2.  If you're currently on Update 13: Create a backup of the hotfix JARs. Open the ColdFusion Administrator. Use the Server Update page to install Update 14. Put the hotfix JARs back. Restart ColdFusion

[Todd in Italy]

Thanks, but one 14 is loaded on my previous install, the admin page gives a 500 error and cannot be loaded to do anything.

Trying your suggestion of removing 14, then updating via the web gui. Removing 14 restores functionality. I cannot use the admin page to perform the update due to security restrictions on the network. I tried giving it the proxy address, but regardless connection to the update site fails with no status code. Installing the patch via java.exe -jar is the only way I can install it.

[Todd in Italy]

I gave up and reloaded ColdFusion2018, the latest install which includes update 13. Setup the IIS connector, the database connection, and some other minor changes to better match production, via the admin page, and the site runs normally. Added update 14 and now get an Apache Tomcat 403 error.  Any idea why adding patch 14 would cause this? I can access the admin page this time, at least. I tried clearing the server cache, no change.

[Todd in Italy]

If anyone knows where this version of Tomcat keeps its log files, that might help. I tried searching for the usual names, like access.log, error.log, but this is a windows install so who knows what they are called.

[BKBK]

Tomcat logs

C:\ColdFusion2018\config\wsconfig\{magic_number}\isapi_redirect.log

C:\ColdFusion2018\cfusion\logs\coldfusion-error.log

C:\ColdFusion2018\cfusion\logs\coldfusion-out.log

[BKBK]

Seems like there's been some movement in the right direction. 🙂

The usual solution to Tomcat 403 errors is to run the Web Server Configuration Tool, \cfusion\runtime\bin\wsconfig.exe

Some more thoughts:

1. Ensure that ColdFusion 2018 is running on a recent JDK, for example, JDK 11.0.15.1
2.  When using command-line Java to install ColdFusion, ensure that the java.exe command belongs to the Java installation that ColdFusion is running on (Cross-check with the setting java.home in /bin/jvm.config)

[Charlie Arehart · Accepted solution]

No, there's no reason doing u14 would itself cause a 403, if you had 13 in place. Let's be clear, you say that you created and tested the connector after the reinstall and before the update, right?

But you say this "latest install which includes update 13". That's not so. The most "recent" cf2018 installer included only update 2.

And it was update 8 that introduced the "ghost cat" remediation, which put a new key into the server.xml. And you need to update the connector after the first update you after that, which will put that key into the worker.properties file. See the update 8 technotes for more on that.

There are still other issues that can bite you from that time, and I have

a blog post from then I could share, but let's see it just updating the connector gets you going. 

[Todd in Italy]

Aha, well I was under the impression that it contained update 13 already. Since it doesn't, then yes I guess I will try rebuilding the connector with wsconfig. 

[Todd in Italy · Accepted solution]

Yes, that did it. Thank you all!

The problem was the way I read the download folder from the licensing site. I thought it referred to the install as 13, implying an already integrated patch 13 which should not have required a connector rebuild. Once I rebuilt the IIS connector via the wsconfig utility the 403 went away and all is working.

Now I can have a good weekend! Next week I'll do the same thing in production!

[Charlie Arehart]

Good to hear, and glad to have helped.

As for the reference to "13" you saw, that's an understandable confusion. Sadly there are some places on the Adobe site where cf version numbers don't use the year (like cf2016, cf2018, cf2021) but instead only the relative number, as was the case with cf11 and before. So these later releases are referred to SOMETIMES as 12, 13, and 14 respectively. It can indeed be confusing and it IS sad to see when that happens. They could fix it, but they do not, for whatever reason.

One last thing: did you save off your prior cf2018 folder, to compare it to this new one, post applying u14? I still think you'll spot a difference in log4j files--which may have names that don't say log4j (like cf-logger.jar) , and may even not be .jar files (if someone renamed that extension thinking it would keep cf/java from loading classes from them, which it may not). As you're on Windows, I'd recommend the tool Beyond Compare (with its awesome free trial--30 days of USE, rather than 30 days from install) as the best for doing that sort of folder compare. 

My point there is that then for prod, you may be able to avoid reinstalling entirely--which is always a risky proposition, as you may forget to take care of or lose something in that effort.

As always, just trying to help.