cf2021 trouble with cfhttp

[stephenb15902562]

I have an interesting issue.  I have installed CF2021 side by side with an existing cf 2016 version in order to evaluate the functionality of all components before releasing it.  When I remove the connectors from 2016 and rebuild them with 2021 all appears to work ok.  We have a nightly reboot of the server scheduled and the next day after the server has come up, none of the sites work and I get an SSL error.

I/O Exception: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

The problem goes away after restarting the coldfusion service but will, and not always, return after a server reboot.   I have removed the connectors from 2021 and placed them back on 2016 except one test site which after every server reboot has no errors in testing the cfhttp or ssl connections by itself.  The only time I have been able to force the problem is by adding additional site connectors back in from 2016 to 2021 and only after the server reboots.

I'm looking for some direction in trying to troubleshoot the problem.   Using JDK 11.0.11 and the proper certifcates for the ssl sites that i'm attemping to hit are loaded into the keystore. 

[Dave Watts]

I don't know how helpful this will be, but that setup you have scares the crap out of me. Setting up and tearing down connectors is basically the most fragile thing you can do with all this. Is there any way you could just have two IIS boxes, one with CF 2016 and the other with CF 2021? They could point to the same shared files for the actual website itself.

Dave Watts, Eidolon LLC

[stephenb15902562]

Dave, 
Thanks for the response.  The process began as a simple migration of CF from 2016 to CF 2021.  Install, migration of configurations, and then moving the connectors.  Testing of all the sites worked until the server rebooted and next morning SSL failures with the error indicated.  Restarting the service fixed it, for that day.  The lack of faith of the stability of my install led me to keeping them up side by side in order to troubleshoot.  3 separate complete re-installs of CF2021 have resulted in the same issue with the SSL error.  I will be the first to admit a gaping lack of knowledge when it comes to these SSL issues.   At this time separate servers isn't an option.

[defaultqgbycw2wjq9a]

In order to get a better idea of what is going on with your SSL issues, what errors are you getting in your log files?

[stephenb15902562]

We have been able to reproduce the problem by moving just one site(connector) over to the cf2021 install.  The test pages and the site worked until after a reboot of the server.  The next morning the test pages that call out via CFHTTP to HTTPS urls fail and so do the secure LDAP tests on that page.  The test site we moved over even though it is an HTTPS url to get to the site only authenticates currently on 389 without secure ldap.  That page works fine and so do all the datasources.  We are currently trying to evaluate the log files with all of the SSL output.  This line has jumped out at me today: javax.net.ssl|DEBUG|20|Timer-1|2021-10-15 04:31:33.356 CDT|TrustStoreManager.java:161|Inaccessible trust store: C:\ColdFusion2021\jdk-11.0.11\lib\security\jssecacerts

That file doesnt exist and I'm not sure where or why that file is even trying to be called from.  The certificates we have imported into the java keystore are in the cacerts file.

I'm sure the log file will reveal more as I decipher it but for now that is the only "error" that I've seen or been able to realize as an error.

[stephenb15902562]

Additionally, when successful the SSL logs show the URLS when they are hit from the test page and go through the handshake, but when the test pages fail the only thing output in the logs is an SSL handshake process to the cf server itself.  Here are some snippets, i'm leaving out the guts of each segment.

Consuming ClientHello handshake message (
"ClientHello": {

Produced ServerHello handshake message (
"ServerHello": {

Produced server Certificate handshake message (
"Certificates": [

Produced ECDH ServerKeyExchange handshake message (
"ECDH ServerKeyExchange": {

Produced ServerHelloDone handshake message (
<empty>

onsuming ECDHE ClientKeyExchange handshake message (
"ECDH ClientKeyExchange": {

Consuming ChangeCipherSpec message

Consuming client Finished handshake message (
"Finished": {

Produced ChangeCipherSpec message

Produced server Finished handshake message (
"Finished": {

onsuming ECDHE ClientKeyExchange handshake message (
"ECDH ClientKeyExchange": {

Consuming ChangeCipherSpec message

onsuming client Finished handshake message (
"Finished": {

Produced ChangeCipherSpec message

Produced server Finished handshake message (
"Finished": {

Received alert message (
"Alert": {
"level" : "warning",
"description": "close_notify"
}

outbound has closed, ignore outbound application data

Closing outbound of SSLEngine

And nothing after that.   

[BKBK]

When I remove the connectors from 2016 and rebuild them with 2021 all appears to work ok. 

By @stephenb15902562

Imagine one of those men in white coats telling you the following, "When I remove the organs from the human ancestor and rebuild them with those of a modern-day human all appears to work ok."

CF2016 and CF2021 are two generations, dozens of updates and hundreds of patches apart. I feel the same as @Dave Watts: "...that setup you have scares the crap out of me."

My suggestion is, however you cut it, make sure CF2016 and CF2021 are two independent applications. As such, neither should depend on the other in any way. 

[Charlie Arehart]

With all due respect to Dave and BKBK, and to Stephen and his interpretation of events, I don't think this is an issue related to the connector's at all.

I mean, of course, switching them from the one version to another is needed and the change of cf versions is key here. I mean the error doesn't to me reflect a problem with having done that in a single machine. And FWIW, that idea should not really be that scary. I do it with people at least weekly. Can there be problems? Sure. And nearly always, they reflect mistakes in following the process (then or in the past) rather than any inherent flaw in the process Adobe has provided. Still, I understand how Gandalf would want to warn, "beware the Balrog ". 🙂 

So onto Stephen's trek through the mountain here: I'm more intrigued by the error he noted, about C:\ColdFusion2021\jdk-11.0.11\lib\security\jssecacerts. That suggests to me something amiss. Stephen, you said "that file doesn't exist". Does the folder? It would not, by default. Instead the jvm within cf by default should for you be at C:\ColdFusion2021\jre. It it that you or a colleague created the jdk-11.0.11 folder there as a sibling to the jre folder? Some take that approach.. Others put the new jvm outside of cf's folders. Either way, is the "Java home" in the cf admin jvm page or cf's jvm.config file pointing to that jdk folder? 

And do you see any other jvm args pointing to it, as those are shown at the bottom of the cf admin jvm page, or in the java.args line?

Something just feels potentially amiss about the jvm setup. Maybe some other tweak made that you're not mentioning. Maybe not. 

To be clear, none of that (nor the connector configs themselves) would explains readily why your problems come and go with cf restarts (of cf2021) ...other than perhaps the dsn lookup caching that the jvm does (which is cleared on a restart). But are you saying, Stephen, that on Every cf2021 restart it fails? Or might things work and then fail on a subsequent cf restart? 

I realize you just want the problem solved, and perhaps without all the questions and theories. I'm just trying to help bound the problem. Solvong complex problems like this here in the forums can be especially challenging, and all the more when many of us are chipping in, each with different perspectives. 

If you're game to follow along, we're all tenacious dogs when a problem "comes into our yard". 🙂 We may resolve it here. 

If instead you "need this done yesterday", some of us here offer consulting, where we could do a screenshare and perhaps find and fix the problem far more readily--finding and following clues in real time, vs via back and forth here. Totally your call. I list myself and others at cf411.com/cftrouble.

Hope we get your problem solved soon, either way. 

[BKBK]

Using JDK 11.0.11 and the proper certifcates for the ssl sites that i'm attemping to hit are loaded into the keystore. 

By @stephenb15902562

What is ColdFusion 2016's update level? Is it using the same Java (that is, JDK 11.0.11)? 

I/O Exception: java.security.NoSuchAlgorithmException: Error constructing implementation (algorithm: Default, provider: SunJSSE, class: sun.security.ssl.SSLContextImpl$DefaultSSLContext)

Java documentation says of the NoSuchAlgorithmException: "This exception is thrown when a particular cryptographic algorithm is requested but is not available in the environment."

The problem goes away after restarting the coldfusion service but will, and not always, return after a server reboot.   

Since the issue is intermittent, that suggests to me it might be due to intermittent lack of access. Possible causes are:

1. The certificate(s) in question is (are) actually missing from your trust store. The error occurs when ColdFusion needs to use a certificate. Just to be sure, import the certificate(s) again.
2. Access to cryptographic files is forbidden. That might be the case if the Java version that ColdFusion 2016 (or 2021) is currently running on were different from the Java version that was in effect when the certificates were installed.
   
   To resolve the issue, ensure that the absolute path of every relevant library is on the classpath. It is easy to find all the lib directories. Just search for lib in the ColdFusion2016 (or ColdFusion2021) directory. In particular, make sure that 
   
   {application.home}/wwwroot/WEB-INF/lib/
   
   is on the classpath. On CF2021, try loading all packages, and see whether that helps. If it does, then we can go on from there.
3. There is a related issue, assuming you are using sandbox security. In your sandbox security settings, enable read-access for the current Java home directory. See, for example, https://stackoverflow.com/questions/52702417/using-cfhttp-to-a-secure-site-fails-when-coldfusion-san... 

[stephenb15902562]

Thank you for all the responses. I am working to get this resolved as soon as I can but its not a yesterday scenario so I highly appreciate all the feedback and suggestions you are all offering. I will try to fill in the gaps of information as much as I can.

CF2016 Update 17
CF2021 Update 2

We always run a different version of JAVA (usually the latest that is supported) in a folder next to the initial JRE of CF. So for CF2021 we point to the jdk-11.0.11.
CF2016 we are currently pointing to jre8u172

We run a test site on cf2021 that has a single page which runs various things like CFHTTP calls and LDAP auths to determine if CF2021 is functioning. Every single night (since the problems occurred) we have been doing a reboot of the server. On ever single reboot the test site functions normally, by itself.
When connectors are removed from CF 2016 and added to CF 2021, the test site and the newly added site functions correctly.
A nightly reboot happens and both services of CF2016 and CF20201 start up and the test page CFHTTP calls fail to HTTPS sites and the Secure LDAP Auth fails. CFHTTP Calls to HTTP addresses and non secure ldap continue to work.

We have never used a jssecacerts file that I can find in any of our previous installs so I'm not sure what error presents. From what I can tell from research online it just defaults to cacerts if the jssecacerts doesnt exist. We utilize the cacerts file.

The JVM configs were likely brought over by the migration process from our cf2016. The only additions we made to the configs after were to add some ssl debugging lines.
-server --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.util.cldr=ALL-UNNAMED --add-opens=java.base/sun.util.locale.provider=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseParallelGC -Djdk.attach.allowAttachSelf=true -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file={application.home}/lib/logging.properties -Dtika.config=tika-config.xml -Djava.locale.providers=COMPAT,SPI -Dsun.font.layoutengine=icu -Dcom.sun.media.jai.disableMediaLib=true -Djavax.net.debug=ssl🤝verbose:keymanager:trustmanager -Djava.security.debug=access:stack

After the initial migration I did alter the server.xml file from the cfusion/runtime/config in order to add HTTPS access to the CFIDE. This was my first time doing anything with the tomcat stuff so it is possible that I messed something up here. Evaluating the file however nothing appears out of the ordinary. We had this added to our CF2016 server to allow our machines to access the cfide securely. I modified those lines previously used with these lines because I was troubleshooting the SSL errors thinking that some of the ciphers may have been out of date and causing the issue. So this is the current line.

<Connector SSLEnabled="true"
ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384" clientAuth="false" keystoreFile="ourkeystorefilehere" keystorePass="changeit" maxThreads="150" port="8453" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true"/>

From what I can tell the class path is full of all the libs. We aren't using the sandbox security.

The hardest part of all of this is trying to wrap my head around why it works and then it doesn't.

Attemping to set the CF2021 application server service to a delayed start had some minimal success but wasnt full proof either.

[Charlie Arehart]

Stephen, I still think we can get you to your solution.

First, though, if your cfhttp calls fail when you connect the web connector to cf2021, the issue has nothing to do with the connector for port 8500...assuming the URLs for your CFHTTPs are not naming that port. That port is for the CF Admin only (or technically the built-in web server for CF--but if your URLs of your cfhttp's don't point to that, they re not using it).

So again: for now don't focus on the connectors. If they are stock, I don't think that's where you prroblem is (not in the ajp connector from IIS, not in the built-in web server connector).

So you've clarified that you are using Java 11.0.11 (you had said it before, but now you have clarified that you put the 11.0.11 in as a sibling to the CF jre folder.)

So first, you've not clarified what java version your CF2016 ran. Second, rather than relying on "knowing the folder" the JVM is, can you confirm what the CF Admin "settings summary" page shows is the JVM used (for both CF versions)? Just a good sanity check.

Second, had you tried running things with CF2021 BEFORE you updated it to use Java 11.0.11? The initial CF2021 installer came with Java 11.0.1. The new installer (from last month) comes with 11.0.11 out of the box. Since you have implemented your own 11.0.11, I am suspecting you had the initial installer and so were at first running 11.0.1. (If you point your CF java.home back to the coldfusion2021\jre and restart CF, that will confirm it--but you could more simply the "java -version" command from the commandline within the coldfusion2021/jre/bin folder.)

The reason I press this last point is that if you had NOT tried your cfhttp with CF2021 BEFORE updating to Java 11.0.11, and especially if your CF2016 was also using an older Java 8 or 11, there was a change in the April updates to Java (8 and 11--291 and 11, respectively) where those and later java versions block calls via https/tls to any server that is not supporting at least  tls v1.2. Your older CF would not have cared, and even the original JVM with cf2021 would not have cared, but 11.0.11 does care.

And the servers you're calling via the CFHTTPs that are failing may well not yet support TLS 1.2 or above.

Note that you can "remove that condom" Oracle has put on Java (and so CF) and allow your server to talk to a "dirty server not yet supporting TLS v1.2" if you really want to. That involves a single line JVM config file change, that was discussed in the technote for those April JVM updates (but few read those). I also discussed it and name the file and arg to change in a blog post I did at the time.

Please let us know if that helps. I appreciate that sometimes long answers are tedious. When we have to guess at things based on what you share, that sometimes means we have to read between the lines and guess also at things you did NOT say. 🙂

Finally, if you confirm this is "not it", then that tells us to look elsewhere. And I'd argue that the ssl/tls debug logging that you show doing in your jvm args would be the next place to go...but  a) you are only telling it to log ssl. If you change it to "all", that will also log tls communications.Even then, b) digging through that dumpster really IS tedious. So I hope the answer I offer above may solve things for you.

[BKBK]

CF2016 Update 17
CF2021 Update 2

We always run a different version of JAVA (usually the latest that is supported) in a folder next to the initial JRE of CF. So for CF2021 we point to the jdk-11.0.11.
CF2016 we are currently pointing to jre8u172

By @stephenb15902562

You said that you want to have a comparable environment on ColdFusion 2016 Update 17 and ColdFu2021 Update 2. To this end, Java 8 Update 172 is out of place. Its TLS capabilities are much weaker than those of JDK 11.0.11. 

For example, Java 11 supports TLS 1.3, whereas support for TLS 1.3 on Java 8 only began from Update 261 b12 onwards.

To cut a long story short, I would suggest you run CF2016 on Java 11.0.11 as well or, at least, to the most recent version of Java 8.

The JVM configs were likely brought over by the migration process from our cf2016. The only additions we made to the configs after were to add some ssl debugging lines.
-server --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.util.cldr=ALL-UNNAMED --add-opens=java.base/sun.util.locale.provider=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseParallelGC -Djdk.attach.allowAttachSelf=true -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file={application.home}/lib/logging.properties -Dtika.config=tika-config.xml -Djava.locale.providers=COMPAT,SPI -Dsun.font.layoutengine=icu -Dcom.sun.media.jai.disableMediaLib=true -Djavax.net.debug=ssl🤝verbose:keymanager:trustmanager -Djava.security.debug=access:stack

Your Java arguments seem OK. However, the forum's display seems to have mangled one of the switches. I was expecting to see something like:

-Djavax.net.debug=ssl‌‌:handshake:verbose:keymanager:trustmanager 

So this is the current line.

<Connector SSLEnabled="true"
ciphers="TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,
TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_RSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384,
TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_RSA_WITH_AES_256_GCM_SHA384,
TLS_DHE_DSS_WITH_AES_256_GCM_SHA384" clientAuth="false" keystoreFile="ourkeystorefilehere" keystorePass="changeit" maxThreads="150" port="8453" protocol="org.apache.coyote.http11.Http11NioProtocol" scheme="https" secure="true"/>

Is it really necessary to specify a list of ciphers. Where or how did you get the list anyway?

In my experience the default list of ciphers - the one automatically used when you omit the ciphers attribute  - is usually sufficient. In other words, it might be sufficient to use:

<Connector SSLEnabled="true"
    sslProtocol="TLS"
    clientAuth="false"
    keystoreFile="ourkeystorefilehere"
    keystorePass="changeit"
    maxThreads="150"
    port="8453"
    protocol="org.apache.coyote.http11.Http11NioProtocol"
    scheme="https"
    secure="true"/>

Note that I have added sslProtocol="TLS"

[stephenb15902562 · Accepted solution]

Thank you all for your continued support in trying to help me troubleshoot the issue. We believe we have solved our problem. Ultimately it seemed like a two part puzzle. The first issue seemed to pertain to the local user account we created for coldfusion service to run under. We had two different versions of coldfusion running on the same service account, some files that were being created by coldfusion, for that local account, seemed to be  getting confused in the same directory by the servers. The second piece of the puzzle seemed to be that Java 11.0.11 defaulted the keystore type to pkcs12 and we needed to manually change that to JKS in the java config file. Changing the local service account and some clean up on the configs seems to be our solution. So thanks again to all your help.

[BKBK]

Thanks for sharing your findings.