New CF2023 Install on Linux Can't get CFLDAP to Work

[flyingmonkey1]

I have a newly setup REHL 9.3-0.5.el9.x86_64 server. I have Apache 2.5.47 running on it. I installed CF2023 using the GUI installer and set it up with the Production Profile. I upgraded to Update 7 and then upgraded the Java to 17.0.11 and in the CF Admin I can see that it is running Tomcat version 9.0.85.0. In Apache I setup two virtual hosts and on my windows machine I edited my hosts file so that I can access both of those sites through a web browser on my local computer.

I can access the CFAdmin just fine and .cfm test pages work fine. I am having problems getting the cfldap tag to work. I have setup many CF servers over the years from CF2008 until the current version and to get LDAP to work all I ever had to do was install some LDAP security certs into my java key store, restart CF, and then it would work. I have done that for my new server as well but I get an error on my test page “One or more of the required attributes may be missing or incorrect or you do not have permissions to execute this operation on the server.” I have worked with the LDAP team and the Unix team and we can verify that LDAP call work using the command ldapsearch in the bash shell but we have tried deleting and reimporting the certs and trying different java installs with the certs imported. Everything we have tried so far we get the same error. The exact same ldap test page I am using on the new server works fine on my other servers so I know the code is correct.

I can’t find anything helpful in the log files. It is basically the same error message:

I have installed the certs into the correct java that CF is using.

Does Java have log files that might give better debug info? Or does anyone have a suggestion?

[Charlie Arehart]

I have a few thoughts for you, and perhaps an unexpected solution.

1) First, Java has no built-in logs to help you, no. There's something you could enable, if this turns out to still be about the tls communication/certs--but I realize you think you're all set there (imported cert, updated jvm).

2) Before digging further, are you saying any of this was WORKING for this cf2023 instance, before you did any of that? Or did you do all that and THEN test things? If it was working BEFORE but then stopped--and that's what prompted the updates--do you know of anything that changed regarding the ldap server you're calling? I realize it may be out of your hands/you have no idea.

3) Next, before enabling any "java logs", you've said you couldn't find anything helpful in the CF logs, but can you please look more closely at the exception.log, and specifically when this error occurs? Once you find the same error (above), look DOWN that log for the nested errors that will apear BELOW it .

You may well find that this IS a problem about TLS and/or  the cert, after all.  Let us know if you find a more specific error (even if you end up solving this), so that future readers finding this could know what more specific error to look for in THEIR exception log.

4) As for why I think it will be a cert problem, it's that someone raised this issue (same error) in another forum thread in the past, and Dave Watts helped them determine that it WAS a problem with how (in their case) they'd experted the cert that they then imported. See that discussion starting at this comment.

5) Finally, if somehow that's not the solution for you, as for enabling other java logging, note that there is TLS handhsake logging you can enable (via jvm args, at CF's startup), and those would be written to CF's coldfusion-out.log file. One form of the JVM arg to add is:

-Djavax.net.debug=ssl,handshake,verbose

For more on using this with CF, see posts such as this one.

But I hope the simpler solutions above may help. Digging through those tls handshake logs can be painf ul.

[flyingmonkey1]

First of Thanks for helping me with this.

To answer your questions:

2) Before digging further, are you saying any of this was WORKING for this cf2023 instance, before you did any of that? Or did you do all that and THEN test things? If it was working BEFORE but then stopped--and that's what prompted the updates--do you know of anything that changed regarding the ldap server you're calling? I realize it may be out of your hands/you have no idea.

Unfortunately, I installed everything and the first thing I did was to update to the latest CF update and the latest Java and then I started running test pages. However, as part of my troubleshooting I imported the certs into the orginal installed Java in the coldfsuion server and switch the JVM to point to it and I get the same errors. That ruled out whether it was a java specific version issue. I have the old server that is still running and working with the same cfldap code and certs but it is running on CF2021 Update 12 and jdk-11.0.21 so it's not exactly comparing apples to apples.

3) Next, before enabling any "java logs", you've said you couldn't find anything helpful in the CF logs, but can you please look more closely at the exception.log, and specifically when this error occurs? Once you find the same error (above), look DOWN that log for the nested errors that will apear BELOW it .

Here is the whole error log from when it starts to the next entry in the log. It looks like gibberish to me but maybe it will give you a clue.

"Error","ajp-nio-0:0:0:0:0:0:0:1-8022-exec-4","05/24/24","10:39:46","","An error has occurred while trying to execute query :us-ldap.nxp.com:636.One or more of the required attributes may be missing or incorrect or you do not have permissions to execute this operation on the server. The specific sequence of files included or processed is: /apps/sites/nww.fab.nxp.com/htdocs/perf_test.cfm, line: 19"
coldfusion.tagext.net.LdapTagException: An error has occurred while trying to execute query :us-ldap.nxp.com:636.
at coldfusion.tagext.net.LdapTag.doStartTag(LdapTag.java:775)
at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:5088)
at cfperf_test_ldap_inlcude2ecfm1326868414.runPage(/apps/sites/nww.fab.nxp.com/htdocs/perf_test_ldap_inlcude.cfm:19)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:256)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:749)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:578)
at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:5088)
at cfperf_test2ecfm1446931643.runPage(/apps/sites/nww.fab.nxp.com/htdocs/perf_test.cfm:31)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:256)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:749)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:578)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:613)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:231)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:209)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at coldfusion.inspect.weinre.MobileDeviceDomInspectionFilter.doFilter(MobileDeviceDomInspectionFilter.java:57)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:51)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:178)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:153)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:168)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:90)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:481)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:130)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:93)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:355)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:450)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:63)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:928)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1794)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:52)
at org.apache.tomcat.util.threads.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1191)
at org.apache.tomcat.util.threads.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:659)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:842)
"Error","ajp-nio-0:0:0:0:0:0:0:1-8022-exec-2","05/24/24","10:49:45","","An error has occurred while trying to execute query :us-ldap.nxp.com:636.One or more of the required attributes may be missing or incorrect or you do not have permissions to execute this operation on the server. The specific sequence of files included or processed is: /apps/sites/nww.fab.nxp.com/htdocs/perf_test.cfm, line: 19"

4) As for why I think it will be a cert problem, it's that someone raised this issue (same error) in another forum thread in the past, and Dave Watts helped them determine that it WAS a problem with how (in their case) they'd experted the cert that they then imported. See that discussion starting at this comment.

I read through his post and I'm not an expert on certs or the order they need to be imported. I imported mine for root down to the lowest level. Maybe I need to delete them and install them in the reverse order and try that. I've never used the openSSL tool or the jXplorer he mentions.

5) Finally, if somehow that's not the solution for you, as for enabling other java logging, note that there is TLS handhsake logging you can enable (via jvm args, at CF's startup), and those would be written to CF's coldfusion-out.log file. One form of the JVM arg to add is:

I added the JVM arg and saved it and restarted CF. After I ran the CFLDAP test page I don't see much in the coldfusion-out log file.

May 24, 2024 15:35:26 PM Information [main] - ColdFusion started
May 24, 2024 15:35:26 PM Information [main] - ColdFusion: application services are now available
May 24, 2024 15:35:26 WARN [main] - Unable to determine dialect of the StAX implementation at jar:file:/home/cimwebadm/opt/ColdFusion2023/cfusion/lib/bundleaxis/wstx-asl-3.2.9.jar!/
May 24, 2024 15:35:26 PM Warning [main] - Unable to determine dialect of the StAX implementation at jar:file:/home/cimwebadm/opt/ColdFusion2023/cfusion/lib/bundleaxis/wstx-asl-3.2.9.jar!/
May 24, 2024 15:35:29 PM Error [Thread-3] - Failed to contact the Adobe Licensing server: {0}
May 24, 2024 15:35:35 PM Error [Thread-3] - Failed to contact the Adobe Licensing server: {0}
May 24, 2024 15:35:35 PM Error [Thread-3] - An error has occurred while validating online.
May 24, 2024 15:35:38 PM Error [ajp-nio-0:0:0:0:0:0:0:1-8022-exec-5] - An error has occurred while trying to execute query :us-ldap.nxp.com:636.One or more of the required attributes may be missing or incorrect or you do not have permissions to execute this operation on the server. The specific sequence of files included or processed is: /apps/sites/nww.fab.nxp.com/htdocs/perf_test.cfm, line: 19
May 24, 2024 15:36:07 PM Information [http-nio-8500-exec-4] - Session rotated successfully.
May 24, 2024 15:36:07 PM Information [http-nio-8500-exec-1] - Starting HTTP request {URL='https://www.adobe.com/go/coldfusion-updates', method='get'}
May 24, 2024 15:36:08 PM Information [http-nio-8500-exec-1] - Connection Failure: Status code unavailable

I think you must have meant the coldfusion-error log because looking there has a lot of info about the certs. The error log for hitting the page once is 598 lines. Instead of pasting all of that text here I'll make a text file of it and attach it. To help you out the certs I'm using have these aliases which are mentioned in the log file.  This is also the order I imported them.

NXPROOTCAG2

NXPInternalPolicyCAG2

NXP-Enterprise-CA4

US-LDAP

[Charlie Arehart]

Yep, coldfusion-error.log, sorry. And this has identified the error. Among all those lines was one clear error:

java.lang.RuntimeException: Unexpected error: java.security.InvalidAlgorithmParameterException: the trustAnchors parameter must be non-empty

And nested below the line with that error is an reference to the specific cfldap call in your cf page. Clearly the error shown in cf was a red herring: this is the real error. 

And if you Google thee the error you find ample discussion of it, definitely about the tls connection from cf (Java) to your ldap. And below is an SO post with lots of ideas to consider. 

But first you refer to having imported certs. How exactly did you do it? Not copying in a cacerts file, I hope. And do you have any non-standard jvm args in the jvm.config?

Then SO post is here :

https://stackoverflow.com/questions/6784463/error-trustanchors-parameter-must-be-non-empty

There will be a solution to this. It seems a procedural issue rather than some bug.Let us know how it goes. 

[flyingmonkey1]

I did a clean import of the the certs. I didn't copy any cacerts files from a different computer. I used this process to import the certs:

1. I copied the certs to the security folder at (/java_for_cf/jdk-17.0.11/lib/security)

2. Then I went to the folder where the keytool is at (/java_for_cf/jdk-17.0.11/bin)

3. And then I ran a command like the one below for all 4 certs I had.

sudo ./keytool -importcert -file /java_for_cf/jdk-17.0.11/lib/security/us-ldap-ha-renew-2024.crt -keystore cacerts -alias "US-LDAP" -v -trustcacerts

The only non-standard JVM arg I have is the one you gave me to log the SSL handshake: -Djavax.net.debug=ssl -Djavax.net.debug=ssl,handshake,verbose

I'm reading the Stack Overflow you sent me do other google searches. I'll try to keep track of everything I do so if I find a fix I can document it.

[flyingmonkey1 · Accepted solution]

I finally got the cfldap to work. I followed some of the suggestions about making sure the keystore type and the truststore type matched. They did and they were both type PKCS12.

Next someone suggested that you could define the path to your keystore so I did that by adding this to the JVM Args:

-Djavax.net.ssl.trustStore=/java_for_cf/jdk-17.0.11/lib/security/cacerts

That did not fix the problem.

Next I added this line to the java args.

-Djavax.net.ssl.trustStorePassword=changeit

And that is when it started to work.

I next removed the jvm arg about the path to the cacerts file since if I ever update java in the future I would have to update that line as well so that makes it one less thing to remember when I update java in the future. After the CF restart it still worked which is good.

So in the end all I had to do is tell java the password for the truststore. I'm not sure why I never had to do that before on linux or windows but maybe the newer version of java, or CF, or RHEL that I'm using requires it. Anyway, I hope this post helps someone in the future solve this problem.

Thanks again Charlie for pointing me in the right direction.