jmx exporter error in coldfusion conatiner

[sirTangale]

i am trying to use jmx exporter in my coldfusion 2021, i have added to the jvm args like this 

-javaagent:{application.home}/jmx/jmx.jar=9101:{application.home}/jmx/config.yaml

it works fine in an standalone ubuntu server, i can see the metrics exported at port 9101, but when i do the same inside docker container it give me error 

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

Exception in thread "main" java.lang.reflect.InvocationTargetException at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:566) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513) at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525) Caused by: java.lang.NoClassDefFoundError: com/sun/net/httpserver/Authenticator at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:39) ... 6 more Caused by: java.lang.ClassNotFoundException: com.sun.net.httpserver.Authenticator at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581) at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) ... 7 more

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""""

the docker conatiner is from official adobecoldfusion/coldfusion2021

jmx exporter is form prometheus 

https://github.com/prometheus/jmx_exporter

final goal ios to use kube-prometheus-stack with coldfusion in k8s cluster

[sirTangale]

* note forgot to mention 
it works when i manyally add a jdk to install jdk eg [ jdk-11.0.17 ] from external source, extending the base adobe/coldfusion image, but can it be done from base image,without the need to modify the base image or custom custom Dockerfile 

[Charlie Arehart]

I've not yet had a chance to test this on my own, but can you try adding these to those jvm startup args for cf:

--add-modules jdk.httpserver --add-exports jdk.httpserver/com.sun.net.httpserver=jdk.httpserver

If it works (or not), please let us know. 

[sirTangale]

thank you for the advice ,
i tried adding the above to jvm args 
but it just stuck saying 
""""""

2024-01-11 12:59:50 Starting ColdFusion 2021 server ...
2024-01-11 12:59:57 There has been an error starting ColdFusion 2021 server, please check the logs.
2024-01-11 12:59:59 [] Checking server startup status...
2024-01-11 13:00:05 [] Checking server startup status...
2024-01-11 13:00:11 [] Checking server startup status...
""""""
and it did not start the coldfusion server , there was no entries in cf logs 
* i think the bare adobecoldfusion/coldfusion2021 does not include jdk it just has jre to run java application 
coz when i install jdk-11 and change java home path to jdk-11 it works [ jmx exporter ] for prometheus 

[BKBK]

Are the foloowing available in the docker installation?

• {CF_HOME}/jmx/jmx.jar
• {CF_HOME}/jmx/config.yaml

[sirTangale]

JMX is an external tool used for monitoring Java applications. It is not available in the Docker image from Adobe. I have added it separately.
https://github.com/prometheus/jmx_exporter

[BKBK]

@sirTangale , You might have misunderstood what I meant. There was no need to explain Prometheus. I am aware of Prometheus. I am also aware of its use in exposing JMX Beans. I shall repeat my suggestion.

You say that you use the Java flag,

-javaagent:{application.home}/jmx/jmx.jar=9101:{application.home}/jmx/config.yaml

However, the name of the Prometheus Jar download is jmx_prometheus_javaagent-0.20.0.jar . So I expected the flag to be something like

-javaagent:{application.home}/jmx/jmx_prometheus_javaagent-0.20.0.jar=9101:{application.home}/jmx/config.yaml

I would also expect the following 2 files to exist:

• {application.home}/jmx/jmx_prometheus_javaagent-0.20.0.jar
• {application.home}/jmx/config.yaml

[sirTangale]

oh thank you,
i have just renamed the jar file jmx_prometheus_javaagent-0.20.0.jar to jmx.jar 
for simplicity 

[BKBK]

i have just renamed the jar file jmx_prometheus_javaagent-0.20.0.jar to jmx.jar 
for simplicity 

By @sirTangale

Ah, OK.

A search on the web shows that what you've found is probably a known bug. It has been observed for Java Agents such as Prometheus:

https://github.com/prometheus/jmx_exporter/issues/850

https://github.com/prometheus/jmx_exporter/issues/865

https://discuss.elastic.co/t/monitoring-es-jvm-with-jolokia-javaagent/352454

If you follow the first link, and scroll to the bottom, you will see a suggestion. The first part is identical to Charlie's suggestion. Since you tried that, and it didn't work, let's put it to one side. (Out of interest, I tested the flag --add-modules jdk.httpserver --add-exports jdk.httpserver/com.sun.net.httpserver=jdk.httpserver on ColdFusion 2023. ColdFusion would not start.)

We are then left with the suggestion about modifying the Java policy. You could try that.

Test by adding the following lines to the /conf/security/java.policy file in the Java installation that ColdFusion uses.

 permission java.net.SocketPermission "*", "listen, accept, resolve";
 permission java.io.FilePermission "/proc/self/status", "read";
 // Path below must match the location of the config.yaml
 permission java.io.FilePermission "/full/path/to/yaml/directory/*", "read";
 permission javax.management.MBeanServerPermission "*";
 permission javax.management.MBeanPermission "*", "*";
 permission java.lang.RuntimePermission "*";

[sirTangale]

Thank you.
I have updated the java.policy file with the statements above. However, when I restart the ColdFusion with Prometheus JMX Exporter, it crashes again.
For now, I am just installing JDK into my ColdFusion container and then using the JMX exporter which works. for now 

[BKBK]

What do you mean by crash? Could you please share the details of what happens and, if there are any, the errors or exceprions.

[sirTangale]

1> i have updated the java.policy file s show in the attachment 
2> i restarted the cf it restarts fine 
3> i added the below line [ prometheus jmx exporter ] to jvm arg  of cf

-javaagent:{application.home}/jmx/jmx.jar=9101:{application.home}/jmx/config.yaml

3> it crashes 
error code

root@0b852107cd84:/opt/coldfusion/cfusion/bin# ./coldfusion restart
Restarting ColdFusion 2021 server ...
Stopping ColdFusion 2021 server, please wait
ColdFusion 2021 server has been stopped
Starting ColdFusion 2021 server ...
Exception in thread "main" java.lang.reflect.InvocationTargetException
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
	at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.lang.NoClassDefFoundError: com/sun/net/httpserver/Authenticator
	at io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:39)
	... 6 more
Caused by: java.lang.ClassNotFoundException: com.sun.net.httpserver.Authenticator
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 7 more
There has been an error starting ColdFusion 2021 server, please check the logs.

[BKBK]

Thanks for the update.  OK, so we get the same exception as you got originally. This made me to have another look.

I have a new suggestion. It is based on the fact that I think we made a mistake when implementing the fix on ColdFusion.

Try the new idea: add the following to ColdFusion 2021's JVM settings and restart ColdFusion:

--add-modules=jdk.httpserver --add-exports=jdk.httpserver/com.sun.net.httpserver=jdk.httpserver

[BKBK]

Try that, first without the java.policy fix. If it still doesn't help, then try it together with the java.policy fix.

[sirTangale]

thanks for reply 
the string

--add-modules jdk.httpserver --add-exports jdk.httpserver/com.sun.net.httpserver=jdk.httpserver

It was earlier suggested by Mr. Charlie in the same thread. I tried it again with and without the Java policy modification, but to no avail, the error persists. The error is the same as earlier.

[BKBK]

@sirTangale , just to synchronize with you. The string you mention is different from the one I suggest. The one you mention contains 1 equality sign (=) whereas the one I suggest contains three.

--add-modules=jdk.httpserver --add-exports=jdk.httpserver/com.sun.net.httpserver=jdk.httpserver

[sirTangale]

oh my miss but i did tried with the string you provided 
tx

[BKBK]

No worries.

Try it with, then without, the java.policy addition. Remember to restart ColdFusion each time. 

[sirTangale]

Sorry if I communicated incorrectly. I did use the code you gave me, but the issue persisted regardless of whether I modified the Java policy

[BKBK]

Thanks for the update. Before we move on to other ideas, please go to the Prometheus JMX-Exporter site and report the bug. Include the stacktrace in your report. It shows that Prometheus JMX makes use of the non-existent package, com/sun/net/httpserver. Also include the following links, which show that others are seeing the same bug: 
https://github.com/prometheus/jmx_exporter/issues/850 
https://github.com/open-telemetry/opentelemetry-java/issues/4192 

Two new suggestions to try:

1.  The JVM flag
   

   --add-modules=jdk.httpserver -javaagent:{application.home}/jmx/jmx.jar=9101:{application.home}/jmx/config.yaml​

   first without, then with, the java.policy addition.
   Restart ColdFusion.
   
   
2.  Google the text 'where to download com.sun.net.httpserver package' and you will find the link https://web.archive.org/web/20090716001955/http://download.java.net/maven/2/com/sun/net/httpserver/h... 
   Download the Jar file and copy it to ColdFusion's /cfusion/lib/ directory. 
   Restart ColdFusion.

[BKBK]

@sirTangale , did either of my last two suggestions help?

If not, here is yet another suggestion. Try the JVM flags:

--add-modules=jdk.httpserver --add-exports=jdk.httpserver/com.sun.net.httpserver=ALL-UNNAMED 

[sirTangale]

thank you for the suggestions i tried both but got same error in both case
for now i will stick to the working method [ to install external jdk, that install required packages to run jmx exporter ]

@BKBK thank you very much 

[BKBK]

Thanks for the update. 

I am confused by " to install external jdk, that install required packages to run jmx exporter". The reason I am confused is as follows. Whatever method you use, you have to install and run the JMX Exporter alongside ColdFusion. You then have to configure the JVM that ColdFusion runs on, by adding the javaagent flag to it.

For example, to run Prometheus with ColdFusion, it is sufficient to:

1.  Copy the Prometheus Jar file and config.yaml file to a directory that is accessible to the JDK that ColdFusion is running on.
2.  Add the Prometheus javaagent flag to ColdFusion's JVM settings.

[sirTangale]

Yes, that bugs me too. The default Java version [that comes with the Adobe image] is Java version 11.0.11 (released on 2021-04-20) LTS, but it doesn't work. There may be some missing packages due to the preference for a minimal image size.

Java version 11.0.17 (released on 2022-10-18) LTS works.

[BKBK]

Thanks for your reply. I now understand the explanation.

Nevertheless, I now think that the issue you face is not as urgent as I thought earlier. I say that because you should not be using Java 11.0.11 on ColdFusion 2021 anyway.  Adobe's general advice is that you should use the latest Java release for ColdFusion. For ColdFusion 2021, that release is currently Java SE 11.0.22 (LTS). That is what Oracle advises, too.

That said, there is a strong possibility that the cause of the issue is environmental. In all likelihood, the Java 11.0.11 in the Adobe image does not have access to the directory {CF2021_HOME}/jmx/ in your installation. I believe that, if you could sort this out, you would solve the problem.