CF 2018 Stability Issues after upgrading to HF-10

[Michael5DF8]

HI Guys,

Wondering if anyone else is experience stability issues after upgrading to CD2018 Update 10?  We host in the cloud a single server CF2018.  It was on Update 2 and rock solid.  After updating to update 4, then update 10 over a 2 week period, we have lost stability in the system.

The server is now requiring a reboot every 3-10 days.

Thinking it was a memory issues, we have 16 GB on our VM, so we upgraded the heap size to 8192 (since it was always maxing at 2048).  No help.

When the system becomes unrepsonsive for our app, the CPU looks as though it is doing work still (not locked, but fluctuating between 1 and 30%).  The Administrator console is typically unavailable.

Today, for hte first time, the CFIDE administration page showed a CF error dump.

Environment:

Windows server 2012R2 fully patched (15 GB, 4 vCPU).  HDD has plenty of resources.  IIS serves the public.

Server Details
Server Product	ColdFusion
Version	2018,0,10,320417
Edition	Standard
Operating System	Windows Server 2012 R2
OS Version	6.3
Update Level	/C:/ColdFusion2018/cfusion/lib/updates/chf20180010.jar
Adobe Driver Version	5.1.4 (Build 0001)
Tomcat Version	9.0.21.0
ava Version	11.0.1
Java Vendor	Oracle Corporation
Java Vendor URL	http://java.oracle.com/
Java Home	C:\ColdFusion2018\jre
Java File Encoding	Cp1252
Java Default Locale	en_US
File Separator	\
Path Separator	;
Line Separator	Chr(13)
User Name	WEB02$
User Home	C:\windows\system32\config\systemprofile
User Dir	C:\ColdFusion2018\cfusion\bin
Java VM Specification Version	11
Java VM Specification Vendor	Oracle Corporation
Java VM Specification Name	Java Virtual Machine Specification
Java VM Version	11.0.1+13-LTS
Java VM Vendor	Oracle Corporation
Java VM Name	Java HotSpot(TM) 64-Bit Server VM
Java Specification Version	11
Java Specification Vendor	Oracle Corporation
Java Specification Name	Java Platform API Specification
Java Class Version	55.0

[carl type3]

Hi Michael,

I am not noticing any CF2018 update 10 stabilty issues myself however I am using it with current Java 11.0.8. Perhaps you would like to upgrade the Java in use from 11.0.1.

What value are you useing for Metaspace?

Regards, Carl.

[Michael5DF8]

Thanks.  Metaspace was set to 192M.  Changed this to 512M.

Was thinking about the java upgrade but this would need to wait for the weekend.  

Strange though that these stability issues only crept in when we started the upgrade process (and subsequent lockdown).

[Michael5DF8]

CF would not restart with this value.  As per article below, I removed metaspace alltogether and will monitor.

https://www.carehart.org/blog/client/index.cfm/2020/2/24/solving_metaspace_errors 

[carl type3]

Look forward to hearing how that goes. 

[BKBK]

> ... the CFIDE administration page showed a CF error dump.

Could you please share the error dump.

[Charlie Arehart]

BKBK, did you see the other comments from the 21st? I'd think the first question for Michael is rather whether the problem was resolved by the removal of the maxmetaspace. Given the discussion, I think that probably resolved it.

But yes, if somehow it did not, it could help to see the error. It could also help to view whatever error lines might also have been written to any of the CF logs at the same time that error occurred, if the cause is perhaps not obvious from the error shown IN the admin.

[BKBK]

Yes, I saw the other comments. My own comment has a different perspective. Whether or not the metaspace solution works, and I hope it does, it would be helpful to the community to document the error for posterity. 

[BKBK]

Hey, I've only just seen that an error message was actually posted! Sorry about that, Michael5DF8.

[BKBK]

The error shown is:

net.sf.ehcache.CacheException: java.lang.NoClassDefFoundError: Could not initialize class net.sf.ehcache.statistics.extended.StandardPassThroughStatistic
at net.sf.ehcache.CacheManager.init(CacheManager.java:425)
at net.sf.ehcache.CacheManager.<init>(CacheManager.java:269)
at coldfusion.security.SecurityManager.initAuthCacheManager(SecurityManager.java:1752)
at coldfusion.security.SecurityManager.getElemStrFromAuthCache(SecurityManager.java:3742)
at coldfusion.security.SecurityManager.getElemFromAuthCache(SecurityManager.java:3857)

Net.sf.ehcache.statistics.extended is a package that is contained in the system library, /cfusion/lib/ehcache-2.10.3.jar. These are all built in. If you cannot find /cfusion/lib/ehcache-2.10.3.jar, then that will be a red flag right there.

To see this, simply unzip the JAR file. You should see within it the class file, /net/sf/ehcache/statistics/extended/StandardPassThroughStatistic.class. 

This error therefore suggests that there was an error during the update process. I would therefore suggest that you uninstall, then reinstall Update 10.

[BKBK]

For the sake of ruling out: you don't use external Ehcache libraries, do you? (We do, and are therefore careful about the order in which classes are loaded). 

[Michael5DF8 · Accepted solution]

Hi guys,

Thanks for your input.  As an update:

• Yes, although an unistall and reinstall was on the table, we have not yet done this;

• We seem to have established stability back into the system with the removal of the metaspace option in the config file.

We have had to add some drive space and change folder drive letters, necessitating server reboot, so the jury is still out, but stability has improved.