Copy link to clipboard
Copied
We have an application that uses CFLDAP, port 636 to authenticate user to Active Directory. We are getting the following error: An error has occurred while trying to execute query :xxx.yyy.zzzz:636.
The server is running CF2021 Enterprise, on a Windows 2016 server
I can get it to work randomly rebooting the server or starting/stopping the CF Application service. It might start working on the second, third, fourth reboot, etc. Once it is working it is fine until monthly patch reboots and the failure process starts all over again. We do have a CF2018 server also on Windows 2016 server and do not have the issue.
Here is what I have tried, all with no long-term luck in fixing the issue:
The error output is not very helpful.
No entries in Windows, Apache or CF logs when the error occurs.
CFCATCH doesn’t provide anything useful
I feel like this is cert related but can’t find anyway to further diagnose the actual error above to provide any deeper details.
Thoughts/Suggestions?
Copy link to clipboard
Copied
Also: I used KeyStore explorer to import the certs. I also installed "CertMan" in CF. Both tools show the domain and DC certs and not expired.
Copy link to clipboard
Copied
I recommend creating a test page with a hard coded ldap call that you can use to duplicate this issue. It'll be a lot simplier if you're able to remove all the variables that probably exist in your current code. On that test page put <cfdump var="#server.system.properties.java#"> so you can verify the java version and location that's being used.
One advantage of a seperate test page is that if you're current LDAPS connection isn't working, but your test page is it'll indicate the issue is probably not with LDAP, but something you're doing prior to that. For example, however you're creating the filter may not be working right and it's generating an invalid LDAP call.
Try using an LDAP client on the windows server. If you're having problems connecting with that it may indicate the issue is not with CF.
You can try adding -Djavax.net.debug=ssl to your jvm.config and then restart coldfusion. I forget exactly which log that will go to, but I believe the coldfusion-error.log. If you don't see anything useful you could try -Djavax.net.debug=all instead.
Copy link to clipboard
Copied
Hi George. I tried your suggestions and still get the same random results. Working with two fresh “dev” servers built to troubleshoot this issue.
I create a simple page with no variables. I added the debug line to the jvm.config file and rebooted. Snippet of CFLAD code attached
This is the odd part – it took 6 reboots to get it to fail. Reboots 1 through 5, the sample page worked fine, authenticated and returned a good dump result. Reboot 6 the failed. Subsequent reboots would result in around 50% failure rate. Both servers had similar experiences.
I have tried the OpenJDK, Microsoft OpenJDK, JRE that was installed with CF and a copy of the JRE from the CF2018 server that works fine.
The coldfusion-error log didn’t have anything useful that I can see, error message attached
I used LDAPAdmin from the server at a time when the sample app was failing and was able to make a connection.
I have imported the certs from the various domain controllers and domain – same issue repeats.
My gut tells me it is something to do with certs but can’t find anything definitive, especially since some reboots the app will work fine.
Also attached the system variables.
I do believe it is something with CF2021 as the CF2018 server does not exhibit this issue
Copy link to clipboard
Copied
Unfortunately, I've ran into several CFLDAP issues that I was never able to fix, so I may not be the best person to help you troubleshoot this. Hopefully one of the more knowledgeable people will hop in with some good advice. My experience has been that if it's a certificate problem I'll get a "PKIX path building failed" error message and it will consistently fail.
Since you're using hard coded values in your cfldap now I think that eliminates it being part of the issue. If it was it should always fail.
Adding the -Djavax.net.debug=ssl to your jvm.config should have caused a lot of output to go to one of the ColdFusion log files. It's been a long time since I did that, but I remember it showing the parsing of the certificates, and the processing of the connection. You could also try -Djavax.net.debug=all, but that will output even more information to the logs.
I doubt this will matter at all, but you could try setting usetls=true on your cfldap command. I believe the default is false. Assuming the LDAP server is setup for TLS that will change how the connection is negotiated.
I'd also recommend taking a look at the server.log file. It's possible that there's something intermittently failing during the startup that might get logged there.
Another log shot, I’m not sure if you’re using CF2021 with or without Update 1. In Update 1 they fixed CF-4205377 “CFLDAP exception shows up in exception.log” You may want to try with and without the update to see if it makes any difference. It seems unlikely it would, especially if your CF2018 installation has Update 11 which includes the same fix, but it might be worth trying.
Good luck with the troubleshooting.
Copy link to clipboard
Copied
Decided to bang my head against the wall some more this evening on this issue before submitting a ticket. I did change the CFLDAP call to use 389 as a test (I know it isn't secure) and it worked. Reverted back to 636 and adding back in the line for secure="CFSSL_BASIC" and it fails. I can toggle back to port 389 and it will work every time. I added the debug line back to the jvm config and get the attached at system startup but nothing is logged during the failed login attempt. I am using incognito mode on both Chrome and FF to minimize any caching issues. Still very puzzled why NOTHING is logged on the failed attempt
Copy link to clipboard
Copied
Nice piece of debugging there, @Dgcotton . Is it perhaps a Java error? That would explain why nothing is showing in the logs. Did you check coldfusion-error.log?
In any case, someone once reported a similar error on Stackoverflow: https://stackoverflow.com/questions/56904682/how-to-properly-use-tls-1-3-cipher-suites-in-hivemq-get...
It was caused by TLS 1.3. Which gives me an idea.
What happens when you restrict the TLS of ColdFusion's Java to TLS 1.2? You can do so by adding the following 2 flags to the java.args property in jvm.config (remember to create a backup of jvm.config):
-Djdk.tls.client.protocols=TLSv1.2 -Dhttps.protocols=TLSv1.2
It requires a ColdFusion restart.
Copy link to clipboard
Copied
I do believe it is something with CF2021 as the CF2018 server does not exhibit this issue
By @Dgcotton
The error message mentions incorrect syntax. So I am wondering whether CF2021 has a problem with the \ in the username. Have you experimented with that?
In any case, if your code works on CF2018 but fails on CF2021, then there will likely be a bug somewhere. You have enough information here to create a bug ticket. Include the attached text files in your bug report, as well as a link to this discussion.
Copy link to clipboard
Copied
In any case, if your code works on CF2018 but fails on CF2021, then there will likely be a bug somewhere. You have enough information here to create a bug ticket. Include the attached text files in your bug report, as well as a link to this discussion.
By @BKBK
Hi @Dgcotton , did you, by any chance? If so, what is the number of the bug ticket? Then we can go vote for it.
Copy link to clipboard
Copied
Have done some more troubleshooting. I worked with our Active Director admin while attempting to authenticate. He shared this logged message from his end:
Internal event: An LDAP over Secure Sockets Layer (SSL) connection could not be established with a client.
Additional Data
Error value:
2148074289 The client and server cannot communicate, because they do not possess a common algorithm.
I have tried adding (and restarting CF) these entries to the jvm.config file:
-Djavax.net.debug=ssl
-Djdk.tls.client.protocols=TLSv1.2
-Djavax.net.debug=ssl:handshake:verbose
-Dhttps.protocols="TLSv1.2"
Also tried adding this to the CFLDAP call:
“usetls=true”
All still result in the same issue.
Any insights from the CF community would be helpful.
Copy link to clipboard
Copied
It looks more and more like either the client or the server is still using the outdated TLS 1.0 or TLS 1.1. To rule this out, do:
Copy link to clipboard
Copied
All of them have DisabledByDefault set to one for both client and server except for TLS1.2
Copy link to clipboard
Copied
Don't recall if I mentioned, I have tried with and without Update 1 applied.
Copy link to clipboard
Copied
Copy link to clipboard
Copied
Thanks. Voted. 🙂
Copy link to clipboard
Copied
I've upvoted your ticket. It seems very likely there's some bug in CF2021 since you can eventually get it to work just by restarting the CF2021 service.
In the ticket comments is this statement "I suspect it's a java issue that you are dealing with and it has to do with connecting to the remote servers by IP address rather than host name." The bug tracker website is showing me that comment is from you, but that doesn't seem to make sense. On the off chance you are trying to connect by IP, the server attribute in the CFLDAP command must match either the subject or the subject alternative name(s) in the server certificate.
Copy link to clipboard
Copied
I am not using IP address, I am using a FQDN that does match the subject or SAN names in the CERT
Copy link to clipboard
Copied
I am having the same issue but with cold fusion 2018. I have tried all the suggestions metionned in this post but no luck resolving. I saw that they closed the ticket on tracker for 2021. Can anyone help with debugging this on cf2018?
Copy link to clipboard
Copied
My test CF21 systems still exhibit this issue. I did take one of the systems that was having the problem, uninstalled CF21 and reinstalled BUT did not apply update #1.
I have rebooted it half dozen times and it is working as it should.
This has to 100% a CF issue. I changed nothing else on the server. Only uninstalled and reinstalled. At some point I will have to re-install the updates on this test box.
Still no understanding why this is happening
Copy link to clipboard
Copied
I've read through all the messages over the past few months here, and I think I can offer some new things worth considering.
1) For example, I see dgcotton saying that it works always if he uses port 389 but haltingly if he uses 636. The latter presumes to use TLS, while the former does not. So first, to BeRadB and George and anyone else who has experienced issues with CFLDAP, can you confirm if you see that same pattern? Whether you do or do not, there's more to consider.
2) Equally important I think, there's something I never see clarified which is EXACTLY what Java version (point release, not major release) is being used with any CF that is failing (or working), whether with CF2021 or CF2018? Can each of you who has the error please confirm, for the failing and working ones?
And I raise this because the Java updates from April on (11.0.11 and 12, or 1.8.0_291 or 301) have an important change where the JVM no longer allows calls out to servers that support only up to TLS 1.1 or 1. (I have a blog post with more on that jvm update.) As such, a cfldap call, or cfhttp, or cfmail, or cfftp, and so on would all now FAIL if the server they are CALLING out to ONLY supports those and not TLS 1.2 or above, once someone applies those JVM updates or later. (And FWIW, I discuss in the post how one COULD modify things to ALLOW again calls to TLS 1.1 or 1.0, if necessary.)
Now, I know that the matter of TLS versions did come up briefly in the thread, and dgcotton said then that he confirmed that "both the client and server" showed supporting TLS 1.2, judging by his assessment of reg keys as suggested by BKBK. (I'll assume that by "server" he means he checked the LDAP server that CF is trying to call, rather than the CF server itself, as that doesn't matter in this case of a cfldap call.)
3) But here's something else to consider, especially given the on again/off again nature of the problem dgcotton saw: what if somehow the request leaving CF is going to some server OTHER than the actual one you checked? What if the ldap is somehow behind a load balancer? If somehow only ONE of those servers was setup for TLS 1.2 and the other was still 1.1 or 1.0, then CF (the JVM) would reject that request if you were on either of those updated Java versions above.
4) In wrapping up, again, please confirm that java version (major and minor numbers) first and foremost--and all the more, dgcotton, please check what it is in the "working cf2018", because if THAT CF's jvm is EARLIER than one of those above, then THAT could be why that CF instance "never" has this problem.
Finally, to be clear, one should not base any assertion of "what the java version is" on what they think they know about what JVM's are or show to be installed in the OS. What matters specifically is what JVM that CF instance points to, as the java home, in the CF Admin. And if the version is not in the folder name, look at the CF admin "settings summary", as that lists the jvm version.
It could really be that this is not at all "about CF2021" but just about the JVM version, and this TLS issue, especially given some of what's been discussed above.
But I'm just going based on that. Perhaps new info will show things to be otherwise. Hope that's helpful.
Copy link to clipboard
Copied
Hi Charlie, I have tried many different versions of the JRE.
I have been working with three different servers, trying different things.
I did take my "test02" server, uninstalled CF21, deleted folders, and did a fresh install - nothing else changed and now it works! I have NOT applied update #1
For the other servers here is all the versions I have tried.
All servers have the same configured path d:\ColdFusion2021\jre
cftest02 - working
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
prod06 - Failing
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
prod06 - Failed
openjdk version "10" 2018-03-20
OpenJDK Runtime Environment 18.3 (build 10+44)
OpenJDK 64-Bit Server VM 18.3 (build 10+44, mixed mode)
prod06 - Failed
openjdk version "16.0.1" 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-24)
OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)
prod06 - Copied jre folder from working test02, rebooted server - still failed
Prod05 - Failing
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
prod05 previous tries
Failed
openjdk version "11.0.11" 2021-04-20
OpenJDK Runtime Environment Microsoft-22268 (build 11.0.11+9)
OpenJDK 64-Bit Server VM Microsoft-22268 (build 11.0.11+9, mixed mode)
Failed
java version "11.0.9" 2020-10-20 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.9+7-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.9+7-LTS, mixed mode)
Failed
java version "11.0.1" 2018-10-16 LTS
Java(TM) SE Runtime Environment 18.9 (build 11.0.1+13-LTS)
Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.1+13-LTS, mixed mode)
Copy link to clipboard
Copied
dgcotton, as for this server that now "works", are you saying it's worked for minutes, hours, days, or weeks? That could be useful to know. 🙂
Also, I see that Priyank has stepped in and is offering to dig through the ssl/tls debug logs (as George had suggested back in June, and you shared only some of the results in one of your attachments). Perhaps if you share more with Priyank he may see something helpful.
Until then, you have responded here with what I can tell is the output of the java -version command, as I gather you ran on each of the servers you were assessing. To be clear, are you sayign you went to that d:\ColdFusion2021\jre folder you mention and ran that there?
I specifically tried to avoid any confusion about things when I asked instead that you (and any involved here) report specifically the jvm verison shown in the CF Admin's "settings summary" page.
Can you report that, for the one that IS working, and for any where it is NOT working? I realize it may seem pointless, but we can't know (and folks could make incorrect assumptions) until it's confirmed.
Copy link to clipboard
Copied
The "working" (cftest02) server has been in a functional state for approximently three weeks now with frequent reboots as that would be when I would see the problem surface. Again, have not applied update #1
Yes, that is correct where I ran the "java -version" command and those versions were in alignment with what displayed on the CF admin console. In every case of making a switch, I always rebooted the server and didn;t rely on stopping/starting the services
For the three cf2021 servers I have here are their present states:
cftest02 - working
prod06 - Failing
prod05 - failing
Note: in EVERY circumstance for the three servers above they all worked after a fresh install. But after period of time of applying CF update and /or reboots they would start failing with NOTHNg else being changed on the server outside of update #1. Reverting Update #1 did not return the server to a functional state
Copy link to clipboard
Copied
OK on your confirming that your "working" server has been fine for 3 weeks, and for now has not had update 1 applied. And you feel that it may cause the problem. Well, you can just give it a try. The update 1 is easily "uninstalled" (no need to uninstall all of CF), which would have it restore (from a backup taken by the update) all the files that the update would change.
Or if you have a reason to doubt the efficacy of the update uninstall's restoring of files, and you fear that after the uninstall of the update things may "still not work", you could yourself save off a copy of the entire CF2021 folder, then do the update. (To be clear, the CF update does not touch files outside of the CF2021 folder.) If you found that somehow after the uninstall of the update, things did not work, you could compare the two folders (your backup with the now-current cf2021) to see what was different.
But most important, if you DID do update 1 and the problem DID NOT recur, then we know now that the problem you have had has NOT been about update 1...and clearly it's NOT about CF2021 itself, as it's working for you on this test02 machine.
As for why you have problems on the other machines, there simply has to be some difference in them. There's been way too much info shared for us to easily piece together (or know for sure, based on what you said) what those differences are, but again since things DO work for a given CF2021 (and you say they HAVE worked in the past for a given CF2018), it seems we're back to there being some environmental issue to explain your times when things do not work...
...that is, unless your CF2021 update 1 now causes things to fail, in which case if reverting makes it goe away, we know there's some issue with update 1.
That said, beware before blaming the update. Sometimes instead what's happened is that there was an ERROR in your applying of the update. That has plagued people on and off since the new update mechanism introduced in CF10. We tend to presume it has "worked" if our admin runs, our apps run, and/or the update shows it's "installed". But there is in fact an update log, and we should always check it after any update, to make sure it shows 0 errors (in a table of errors and successes near the top). I have a blog post with more on that.
Let's see if this helps you move the ball down the field, or again now that Priyank is helping you perhaps he will see something we have not discussed.
Copy link to clipboard
Copied
Hey Charlie,
the 389 port works but is send the service account username and password in plain text
if i switch to port 636 and add the basic secure param it fails every time.
here is the version i am work with.
Server Product ColdFusion 2018
Version 2018.0.11.326016
Tomcat Version 9.0.41.0
Operating System Windows Server 2012 R2
OS Version 6.3
Adobe Driver Version 5.1.4 (Build 0001)
Java VM Version 11.0.1+13-LTS