CF21 FastMethodInterceptorDelegatorBoot class broken by AppDynamics (ClassNotFoundException)

[Bill Tudor]

We have found that when an AppDynamics agent is running in the JVM our scheduled taks fail to run

error something like this:

[ERROR] runwar.context: java.lang.NoClassDefFoundError: com/singularity/ee/agent/appagent/entrypoint/bciengine/FastMethodInterceptorDelegatorBoot

[ERROR] runwar.context:             at coldfusion.cache.ehcache.GenericEhcache$CacheCleanUpAgent.run(GenericEhcache.java)

[ERROR] runwar.context:             at coldfusion.scheduling.Scheduler.run(Scheduler.java:245)

[ERROR] runwar.context:             at java.base/java.lang.Thread.run(Thread.java:829)

[ERROR] runwar.context: Caused by: java.lang.ClassNotFoundException: com.singularity.ee.agent.appagent.entrypoint.bciengine.FastMethodInterceptorDelegatorBoot not found by scheduler [19]

It seems others (including Atlassian and Oracle) have encountered similar when implementing AppDynamics apm, and the accepted solution seems to be a variation of adding "org.osgi.framework.bootdelegation=com.singularity.*" to either JVM or properties files depending on your stack.

Cannot find an equivalent 'place' to add something like this to CF and was wondering if anyone has encountered this and found a solution in CF. Note that CF2016 played nicely with AppDynamics, so this has manifested only after an upgrade to CF2021 and therefore Java 11 and a later version of AppDynamics.

[Charlie Arehart]

Assuming that desired jvm args is the right solution, you ask where to put it. That depends on how cf itself has been deployed. 

1) The place to add such jvg args in a typical cf deployment would be either a) the cf admin's "java and jvm" page and its Java args field, or b) the jvm.config file (found in cfusion/bin or [instancename]/bin) and its java.args line.

Always be very careful editing either of those (changing one changes the other), as a mistake can keep cf from restarting. To be safe, make a backup of the file before touching it.

2) That said, your error message mentions runwar, and that suggests you may using Commandbox to run cf. Is that so? If so it's important to clarify. It offers still other ways to effect a JVM change, though the cf admin page above would still work. 

3) Or you could be deploying cf in yet another way where you're deploying it as a war file, like on Azure App Service or your own implementation of Tomcat or another app server. In that case, the admin does not offer jvm page and there is no jvm.config file--because all jvm arg definition is dependent on the specific app server platform/servlet engine. Again, you'd need to clarify which you're using as there are many variants. 

4) Finally,you could be deploying cf as a container/image, in which case the Adobe cf images offer no means to easily modify the jvm args. But I show an example of doing that among my many sample config files at my "awesome cf compose" GitHub repo. 

5) As for your error, if your arg does not indeed solve it, note that it may not be about the cf version upgrade but instead (as you mentioned) the jvm upgrade which was implicit in the cf upgrade. 

Cf2018 and cf2021 run on Java 11, and earlier versions could depending on cf updates applied, otherwise they ran on Java 8 or earlier. (I have a blog post with a table of what cf versions/updates support what jvm versions.)

6) And as you may know Java 9 introduced a new modularity control mechanism, which defaults to limiting what Java libraries/classes can be called from within Java (and thus cf) code--in the name of greater control/security. 

And that jvm version (and later) added new jvm args to control/override that default control of library loading, such as --add-opens, --add-modules, and --add-exports. Some may have noticed that Adobe has implemented those in the jvm args for cf versions whose installers were created to support implementing versions beyond java 8.

And you may therefore need to add one of those, but the error message you offer doesn't indicate that directly, so again I just offer this for consideration if the above answers to your question about "where to put such args" doesn't get you going.

Let us know how it goes. 

[Bill Tudor]

Update: we have both commandbox and silent Windows installs of coldfusion using the standard msi installer and we have tried both the in-built Cf jre and pointing to a different java_home. All flavours exhibit the problem. We also have other JVM arguments that work as expected and we haven't stopped Cf loading and running. It's just that once the AppD javaagent is running CF can no longer run scheduled tasks as it cannot load dependent com,sigularity.* classes (it's as if the AppD agent has overritten Cf's com.singularity classpath and we see exceptions like posted above 'NoClassDefFoundError'). This only adversly affects the scheduler, even though we also see similar errors in coldfusion-error.log  for mail and ehcachce packages, they still seem to work ok)...so we contacted AppDynamics support....

After a v.long and in-depth call with Appynamics it looks as if the solution that works for other organisations will not work with Cf because any JVM argument to tell apache.felix framework where bootdelegation com.sigularity classes are (or where they are not) is ignored - usually it's a flag something like (in Atlassian's case)

-Datlassian.org.osgi.framework.bootdelegation.extra=com.singularity.*

Their advice "For other OSGI-based runtime containers, add this package prefix to the appropriate OSGi configuration : 

file.org.osgi.framework.bootdelegation=com.singularity.*

Also didn't make sense in context of Cf - as 'where' would we add such a property.

So I was trying to ask how/where can add this fix for Cf; either as a JVM flag or a jar property file.There appear to be no .properties files under the hood of Cf (we exploreed the jars mailnly in cfusion\lib). Hence my vagueness as to 'where' to add this fix. It is definitely a JVM 11 issue as you hint at -  AppD older version JAVA 8 agent jar worked fine with CF2016 (and no, we cannot use the old AppD agent.jar unfortunately - that would be too easy!). 

Not sure it's relevant but we use IIS or whatever Tomcat/Catalina comes out-of-the-commandbox - what I mean is nothing unusual in terms of  application or webservers.

[Dave Watts]

I'm pretty sure you'll need to add this to jvm.config. I don't know what CF uses for "file.org" in the string here:

file.org.osgi.framework.bootdelegation=com.singularity

Anyway, follow up with the CF support team as Abhishek recommends. Good luck!

Dave Watts, Eidolon LLC 

[Bill Tudor]

that string is just taken from AppDynamics suggestion for a 'fix' for 'Other osgi frameworks' - agree it's not clear what CF would do with it or where it would be best-placed; I guess that's at the heart of the problem really.

I got it from this AppDynamics page: https://docs.appdynamics.com/appd/23.x/latest/en/application-monitoring/install-app-server-agents/ja...

[Charlie Arehart]

Bill, the solution would seem to be your needing to add this to your cf jvm config:

-Dorg.osgi.framework.bootdelegation=com.singularity.*

That should be added to the cf admin Java args field (if your cf deployment offers a "Java and jvm" page) or to the jvm.config file for a normal cf deployment (in the cfusion/bin folder).

If you say you do it and it's "ignored", leave off the dash and you should find cf will not start (or stop). If that does not happen, you're not editing the right config.

There are SO Many things that could explain that. I offered several details above to try to help. If you "just need to get it done", and neither Adobe's offer of support nor more help here leaves things unresolved, grab a 15 min slot in my online calendar link offered at carehart.org/consulting. If the time is not valuable, it would not be billable. 

[Bill Tudor]

Charlie, I validated that I am editing the right jvm.config and I am. I may well try and grab a slot in your calendar - thanks for offer. At the moment trying a Adobe support hotfix, but not looking right yet. And yes, a lot of folks seem to find that JVM argument helps (and it is now in the jvm), but so far it's not making a difference (certainly with commandbox); but I am trying in a bog-standard CF2021 install also (Windows, iis)...and unfortuately, yes we do need to 'just get it done' ;o)

[AbhishekJha]

Hi Bill,

Could you please share the ColdFusion2021\cfusion\logs - coldfusion-out, coldfusion-error and exception logs to cf.install@adobe.com

Thanks,

Abhishek

[Bill Tudor]

Thank you Abishek - I have sent logs as requested

[Bill Tudor]

Attention Abishek: please could you provide that hotfix jar again via a secure link (or resend) it appears corrupted as attached on email deleivered to us Tuesday 26th Sep. We think it will work from what we see but believe it is not loading due to corruption. Many thanks for swift response on this.

[Bill Tudor]

Many thanks Abishek - the fix you provided for us is looking good. Thank you for quick turnaround on this.