CFLDAP timeout not working and JNDI.properties

Explorer ,
Mar 25, 2021 Mar 25, 2021

Copy link to clipboard

Copied

How can I set the "com.sun.jndi.ldap.read.timeout" value, or otherwise get a CFLDAP call to properly timeout.

 

I have an Active Directory (AD) server that I'm making CFLDAP calls to.   Randomly the calls will hang for approximately 900 seconds.  The AD server admins say they don't see any issues on their end, but the 900 seconds does correspond to a default MaxConnIdleTime setting on the AD server.   In FusionReactor it will show that it's stuck on "java.net.SocketInputStream.socketRead0"

 

This page matches with the issue I'm seeing:  java.net.socketinputstream.socketread0 Problem patterns 

 

According to the various pages I've found on this issue I need to set the "com.sun.jndi.ldap.read.timeout" value.   I've tried setting this in the jvm.config, but I found web pages that say you can't set it there.  That it needs to be set in the jndi.properties file.  All of the web pages I find for ColdFusion and jndi.properties are from CF9 and older, those paths appear to no longer be valid.    I've tried JAVA_HOME\lib and runtime\lib.   Neither worked, but I'm not sure how to verify that the jndi.properties file is even being loaded.

 

I also tried setting it on the calling page with

<cfset system = createObject("java", "java.lang.System")>

<cfset system.setProperty("com.sun.jndi.ldap.read.timeout","1000")>

but that doesn't appear to work, when I do system.getProperties() I can see it set there, but I suspect I'm not setting it right in the first place.

 

I do have the timeout value set on the cfldap call, and if I set it for a couple milliseconds I can force timeouts.   The timeout value is being ignored though for these random hangs.   The issue isn't with the specific LDAP call, because the exact same call reran will work fine.

 

I'm open to any suggestions anyone has for preventing the CFLDAP call getting stuck on "java.net.SocketInputStream.socketRead0"

TOPICS
Advanced techniques

Views

287

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 27, 2021 Mar 27, 2021

Copy link to clipboard

Copied

I think it is incorrect to set it as an environment property of your operating system. As far as I know, "com.sun.jndi.ldap.read.timeout" pertains to the custom environment that you yourself create for LDAP authentication. 

 

The relevant part of the Java code is:

 

// Set up the environment for creating the initial context
Hashtable<String, Object> env = new Hashtable<String, Object>();

// Set up the properties of the environment
env.put(Context.INITIAL_CONTEXT_FACTORY, "com.sun.jndi.ldap.LdapCtxFactory");
env.put("com.sun.jndi.ldap.read.timeout", "1000");
env.put(Context.PROVIDER_URL, "ldap://localhost:2001");

 

See the LDAP Simple.java example

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

Thanks for the response.  I'd prefer to stick with Coldfusion instead of rewriting everything as Java code, but I might have to go that route if I'm unable to find the proper place to put the jndi.properties file.

 

Looks like someone reported this as a bug against Lucee back in 2018.   I'll do the same on adobe's site.  https://luceeserver.atlassian.net/browse/LDEV-1860 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

George, I have not hit this problem (and can't recreate it), but I have a couple of suggestions that may work for you.

 

1) First, you say you "tried setting this in the jvm.config, but I found web pages that say you can't set it there." So first, how did you try to set it? According to this page on ldap/jndi configuration, it does discuss that particular timeout and later in the doc it shows you CAN set such jndi.ldap settings there. It shows an example of setting other related ones. Is this what you tried?

-Dcom.sun.jndi.ldap.read.timeout=1000

 

(BTW, note also that the doc I shared indicates that this timeout is in milliseconds, so 1000 would be only 1 second. Is that what you were wanting? I might suggest giving it at least a few seconds, as things can happen on any server randomly to slow it down a little, even an ldap server.)

 

2) And are you saying you added that to the java-args in the jvm.config file of the instance in which you were trying things? I'd recommend you try it there rather than in the CF Admin itself, because if you make any mistake you may find CF would not come up (and you could not then see the Admin to correct your mistake).

 

On the other hand, you COULD use the CF Admin as a checkpoint after you make the above change and restart CF. Then in the admin it SHOULD show put in the java args textarea on the CF Admin "java and jvm" page. If you somehow do NOT see it there, then you are NOT editing the right jvm.config. (If you are running an instance of CF, be sure to edit the one in the instance's bin/jvm.config, rather than the cfusion/jvm.config, for example.)

 

3) You say also that you tried to set this in a jndi.proporties file, but you were stumped as to where to put the file. You said you "tried JAVA_HOME\lib and runtime\lib". Can you be more specific? Those can be translated into various locations (by you or by us reading your words), and there are indeed various lib folders within CF. 🙂

 

More specifically, did you try it in the cfusion/lib folder, right under your main CF folder? Or if you are running an instance, then its sibling instancename/lib folder? Those are the folders where one would modify the various jars related to JDBC drivers that CF loads, for instance).

 

And then you restarted CF, and then did your test, and it didn't help? There could be still other reasons the properties file didn't work, if indeed you did put it there. (The Java doc I linked to made no mention of a jndi.properties file.)

 

So I am inclined to think that the first approach above should work.

 

4) Finally, you mention how the cfldap timeout attribute did not help. I will note  that this has been a problem also with cfquery's timeout, cfhttp's timeout, and so on. Folks find that it "doesn't work". I think the problem is in what the timeout is intending to do (but the CF docs are not as clear as they could be).

 

Again, if you look at that Java document I linked to, you will see that besides the read.timeout you are trying to set, there is a VERY similar-looking com.sun.jndi.ldap.connect.timeout, which controls how long the JVM should be willing to wait to obtain a connection to the jndi server. I suspect you're getting a quick connection, but then you are not getting any response.

 

And I won't be surprised if the CFLDAP timeout is instead just about a connection timeout (setting connect.timeout) rather than a response timeout (read.timeout).

 

So let us know if you can try the options above and if this gets you going (and I hope you may try it before writing a tracker ticket to Adobe. The more info you have, the more useful that ticket would be for Adobe and future readers).


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

Charlie, thanks for the reply.
1) This was one of the pages I saw that states it can't be set in the jvm.config:
"The simple answer is that JNDI properties are not system properties, so setting them as system properties has no effect. You can provide them either in the environment of a Context or else via a /jndi.properties file in the root directory of your JAR file."

 

Providing them in the environment of the Context is what BKBK suggested, but that would require me to essentaily write my own java LDAP function and not use CFLDAP. That's why I was hoping to figure out where to put the jndi.properties file.

 

I'm trying the timeout of 1000, to increase the likelyhood of getting timeouts and to know that it's working, I probably should have set it even lower.

 

2) The java-args were added directly to the jvm.config file and CF was restarted. We verified the right jvm.config file was modified. It was also verified, because our CF instance didn't come up the first time due to a typo.  I saw various pages where some said you can set it the jvm.config and others say you can't. So we went ahead and tried it without any luck.   Of course I'm only assuming it didn't work because we still get hung connections and no timeouts.  Do you know a way I can output the jndi properties from within ColdFusion?  I'll see if I can figure that out.

 

3) JAVA_HOME would be the path where the admin had our upgraded version of Java installed \Program Files\Java\jdk11.  We're on version 11.0.10.   The runtime\lib would be \Coldfusion2018\cfusion\runtime\lib. This is actually being done by our server admin so I'll need to have him check if there's a lib folder under the instance. Unfortunately I don't have direct access and it's a shared environment, so everytime we reboot it to test, it stops a bunch of people from working.

 

This is an example Coldfusion MX document that mentions the jndi.properties file is located at cf_root/runtime/servers/default/SERVER-INF/jndi.properties.

This page talks about how the jndi.properties is used to configure the jndi properties of a java application.

4) Based on FusionReactor and the packet capture with wireshark it appears the connection happens quickly, but gets hung during the communication. It appears that CF gets the response back from the AD server, but is expecting more so the connection stays open until the AD server closes it at the 900 second idle timeout. FusionReactor is showing that the connection is hung on the "java.net.socketinputstream.socketread0". If I set the timeout attribute on the cfldap tag to something really low I can generate timeouts but it's these socketRead0 ones that ignore the timeout.

 

I've already opened an Adobe bug report since I saw Lucee had the same bug reported and is implementing a fix for it.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

Sorry, I had a client call so couldn't respond to George's comment before I did BKBK's (my disagreement reply below). I just felt that was important to get out first, while I had time. So now, back to George's reply to me.

 

To your point 1, I see you quote a stackoverflow topic saying these cannot be set in the JVM args. FWIW I quoted an Oracle Java doc that implied that they can. 🙂 I could only go on what I saw and shared. So again, I would recommend you have your admin try it.

 

As for your point 2, you say you "did try", but I can't tell if you're saying you tried exactly that arg as I proposed it. If you did, and the request still hung, that's indeed a bummer to hear. As for how to print out the jndi properties, I don't know that off the top of my head.

 

As for your point 3, no, the JVM location is NOT where to put it (or not where I was proposing to put it). Again, I was sharing from prior experience that the way to modify things loaded by CF was to put them into the cfusion/lib folder (or instance/lib, if running instances), and no not runtime/lib. Again, there are indeed many lib folders where you may find people propsing things over the years.

 

Indeed, your next point offered links to CFMX docs. To be clear, MX was CF6, from 20 years ago--when CF ran atop JRun, and the folder structure changed dramatically in CF10 and above, so do be VERY cautious about following any recommendations from such an old doc (even if the "update date" on the page suggests Adobe has blessed it. That would be in the context the version it was written for. For example, that article refers to the Verity K2 server, which was removed in CF9.)

 

As for that Java 11 doc on configuring JNDI settings, that confirms that the jndi.properties should be put in the classpath. In CF's case, that's a lot of possible folders, but again the key one is the cfusion/lib. (You can see all folders in the classpath in the CF Admin "settings summary", if you wanted to. That will NOT show if it pulls in a specific FILE from that path, but if a file is in a folder in that path, it can be expected to be loaded.) That said, there are matters of order of classloading that can sometimes complicate even that. But try it and see how it goes.

 

As for your point 4, you're confirming what I said: that it's NOT a "connection" issue but a read issue, so again getting the read.timeout should be the ticket.

 

Finally, about the Tracker ticket, ok. And you may well want to wait to see if/when you get any response there. But I am inclined to think that we still have not exhausted our efforts here and may solve it before they would. (And if anything, your ticket reads like you want them to change how the timeout attribute works. I doubt that will be done, since it would conflict with its current use, which I suspect is to set the connect timeout. That's not what you want, but others may want it, so if anything they'd need to consider adding a NEW attribute.)

 

And as for the Lucee ticket, I had read it before I wrote my last reply to you. While I agree that Pete (who wrote it) is indeed asking about the same issue as you, I'll note that the responses have not been about addressing that but instead talking about the current timeout that they DO see--which I am betting is again a connection timeout rather than a read timeout--and about whether their doc should show it being seconds or ms. Perhaps in time they may take it the direction you want, though that would only be for the sake of Lucee of course, and won't help you.

 

That's why I am pressing for you to be sure to try the things I've proposed. 🙂 As always, just trying to help. It's not like I get paid to write such detail. It's because I find often that other resources people rely on can be misleading. I simply want to present as much detail as seems needed. (I know it's more than some want to see or read--not referring to either of you, necessarily.)


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

@Charlie Arehart : 

 

-Dcom.sun.jndi.ldap.read.timeout=1000

 

Charlie, @George Alsobrooks tried that and it didn't work. Not surprisingly. The above flag applies to a JVM system-wide setting. Whereas, what we're looking looking for is a  timeout setting specifically for the LDAP environment.

 

@George Alsobrooks : I'd prefer to stick with Coldfusion instead of rewriting everything as Java code

 

Fair enough. Let's go back to the very beginning of this discussion. 

 

The setting com.sun.jndi.ldap.read.timeout=1000 has more or less the same effect as <cfldap timeout="1000">. However, you say that <cfldap timeout="1000"> doesn't have the desired effect. So, what about omitting the attribute altogether? That will force ColdFusion to use the default value, namely, 5000 milliseconds.

 

Suppose that that still doesn't work. Another thought follows.

 

You say the server's MaxConnIdleTime does have the desired effect. But it times the connection out after a long 900 000 milliseconds.

 

This suggests that the timeout we should be thinking of is the connection-timeout, not the read-timeout.

 

@George Alsobrooks : I might have to go that [Java code] route if I'm unable to find the proper place to put the jndi.properties file.

 

Then probably the proper place  to put the JNDI properties is /cfusion/runtime/conf/server.xml: as an implementation of JNDI Directory Realm (org.apache.catalina.realm.JNDIRealm) in Tomcat.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

I disagree with much you have said here, BKBK. But to save everyone's sanity and time I'm not going to bother to address each point. I stand by what I said (which also detailed WHY I said it and why it seems correct).

 

And I still think the answer is there: either a) in putting in the specific jvm arg I offered (he does not say yet he tried that, specifically) or b) in putting the jndi.properties in the cfusion/lib folder...not any of the others named.

 

Again, it strikes me this is NOT a connection timeout but instead a read timeout. BKBK, if you really might want to contend with what I said, please read all I wrote carefully. Too much of what you say reads as if you did not.  But this may be another one of those topics on which we will simply be butting heads.

 

George, I know you said you're hands-off the server. I hope you will please report if and when you can have your server Admin try the specific things I proposed.


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

Charlie I read everything in your post. The points I make go in a direction different from yours.

 

@Charlie Arehart : I still think the answer is there: either a) in putting in the specific jvm arg I offered (he does not say yet he tried that, specifically)

 

No, Charlie. The setting 

-Dcom.sun.jndi.ldap.read.timeout=1000

is unlikely to be the answer. As I said before, this flag is a system-wide setting. Whereas, what we're looking for here is a  timeout setting specifically for the LDAP environment.

George has in fact tried the setting you mention. Without success. He said,

  • "I also tried setting it on the calling page with
    <cfset system = createObject("java", "java.lang.System")>
    <cfset system.setProperty("com.sun.jndi.ldap.read.timeout","1000")>
    but that doesn't appear to work."

 

@Charlie Arehart : b) in putting the jndi.properties in the cfusion/lib folder...not any of the others named.

 

I saw that suggestion. I ignored it because I don't think it would work. Just instinct. Jndi.properties consists of runtime properties, whereas the files in /cfusion/lib consist of properties for setting up the application. In any case, I know of no basis or justification for placing jndi.properties in /cfusion/lib. But I am willing to learn. 🙂

 

@Charlie Arehart : Again, it strikes me this is NOT a connection timeout but instead a read timeout. 

 

I thought so, too. I turned later towards connection timeout for the following reason. George says the connection lasts for 900 000ms and <cfldap timeout="1000"> fails to break it. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 29, 2021 Mar 29, 2021

Copy link to clipboard

Copied

Thanks for your patience, but we are indeed approaching things from different directions. And I still think I might swing you to my position. 🙂

 

1) First, you say again that the jndi.ldap.read.timeout "is a system-wide setting. Whereas, what we're looking for here is a timeout setting specifically for the LDAP environment.

 

Do you mean to say that you think that this setting would be controlling not Java (not CF) but would be controlling instead the destination LDAP "environment"? My operating assumption is that BECAUSE there is that MaxConnIdleTime (he referred to) set to 900 on the LDAP server, it is THAT which causes the request in CF to finally after awaiting that 900 seconds. I don't expect there's anything on the CF side that can control that (the 900 second timeout). 

 

And that's the purpose of trying to set the jndi.ldap.read. timeout. That goal is not to influence "how long the LDAP server will be willing to wait to serve a response", bur rather to affect how long *CF* (and the Java underlying CF) is willing to WAIT for that response. 

 

And THAT is also why I think this is NOT a conneciton error, because again he DOES get a connection. He even set the cfldap timeout to a small amount and it made no difference...because there was no connection timeout. And tt's the READ timeout which has NO DEFAULT if it's not set, like I said also can happen with CFHTTP and other CF features that call out to other servers.

 

And of course I realize that you (and other resources he's read) have been trying to set an env var instead. I do NOT think that's the only way (and I'm NOT surprised that its proven hard to do, programatically).

 

2) As for the jndi.properties, and yourr argument that it did not seem it will work, here again you make a distinction that this arg is about "runtime properties, whereas the files in /cfusion/lib consist of properties for setting up the application". I really don't see the difference between "runtime" and "application" that you state, otehr than to wonder if it is related to the first point which I have tried to clarify above.

 

FWIW, I have said from the beginning that I had no confirmed experience on any of this. But again I have been reading and quoting the JAVA docs that talk about these things, and then I am associating what THEY say with how I understand things to work in CF.

 

So now we wait, for him to confirm whether EITHER approach works (the properties file or the jvm args). If it does, we will ALL have learned something.  🙂


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Mar 30, 2021 Mar 30, 2021

Copy link to clipboard

Copied

BKBK and Charlie, I appreciate the time both of you are spending trying to help me with this issue.

 

On my end the thread is stuck on socketread0.  I'm assuming that I need to set the read timeout because of the java.net.socketinputstream.socketread0 page that I posted earlier.  On the LDAP server's end it's idle connection timeout is functioning correctly and it's closing the connection around the 900 second mark. I'm looking for a way to tell the CF server to not wait indefinitely in the socketread0 state, I'm not trying to change the LDAP server in any way.  If there's a better way to stop CF from waiting indefiintely I'm willing to try that.

 

Something else I tried was launching the CFLDAP calls within a CFTHREAD and then trying to terminate the threads, but CF wasn't able to terminate the threads and they stayed stuck for the 900 seconds.  If I try and manually kill the request in FusionReactor it fails and stays stuck for the 900 seconds.

 

"-Dcom.sun.jndi.ldap.read.timeout=1000" is what we put in the jvm.config file and there was no change in behaviour.

 

I tried running the cfldap calls without a timeout specified and that didn't help, I still ran into the hung connections.

 

For the lucee bug report I had looked at the git and it shows them adding in "env.put("com.sun.jndi.ldap.read.timeout", timeout);" to the LDAPClient.java file.  I don't expect to get a solution from the Adobe Bug report, at least not one that will help me in the next year or two. I also don't think it would conflict with it's current use. If a user sets a timeout they expect the function to actually timeout, if anything setting the read and connect timeout would help meet user expectation, versus the current situation where you set a timeout and it doesn't actually timeout.

 

I looked at the JNDI Directory Realm document, but I don't see how I can make use of that. The server.xml allows you to create datasources, but CF does not allow LDAP datasources. I didn't see aything in the guide that allowed me to set a property that would apply to all LDAP calls. So I can create the datasource within the file, but there's no way for me to tell CF to actually use that datasource.

 

The server admin I'm working with is out till Thursday, but when he's back I'll try the jndi.properties file under instance/lib to see if that makes any difference.  The jvm.config file also states that "java.class.path={application.home}/lib/oosdk/lib,{application.home}/lib/oosdk/classes" so we'll also try that if instance/lib doesn't work.

 

If it helps this is the top of the stack trace from FusionReactor for a hung connection.

java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.socketRead(SocketInputStream.java:115)
java.net.SocketInputStream.read(SocketInputStream.java:168)
java.net.SocketInputStream.read(SocketInputStream.java:140)
sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:478)
sun.security.ssl.SSLSocketInputRecord.readFully(SSLSocketInputRecord.java:461)
sun.security.ssl.SSLSocketInputRecord.decodeInputRecord(SSLSocketInputRecord.java:243)
sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:181)
sun.security.ssl.SSLTransport.decode(SSLTransport.java:110)
sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1359)
sun.security.ssl.SSLSocketImpl.waitForClose(SSLSocketImpl.java:1620)
sun.security.ssl.SSLSocketImpl.closeSocket(SSLSocketImpl.java:1596)
sun.security.ssl.SSLSocketImpl.shutdown(SSLSocketImpl.java:1577)
sun.security.ssl.SSLSocketImpl.bruteForceCloseInput(SSLSocketImpl.java:689)
sun.security.ssl.SSLSocketImpl.duplexCloseOutput(SSLSocketImpl.java:553)
sun.security.ssl.SSLSocketImpl.close(SSLSocketImpl.java:485)
com.sun.jndi.ldap.ext.StartTlsResponseImpl.close(StartTlsResponseImpl.java:276)
coldfusion.tagext.net.LdapTag.closeConnections(LdapTag.java:870)
coldfusion.tagext.net.LdapTag.doStartTag(LdapTag.java:768)~

 

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Mar 31, 2021 Mar 31, 2021

Copy link to clipboard

Copied

So much there, but let me respond briefly and then add add some new considerations.

 

1) So first, let me just summaruze replying to your last note by saying I hear you on all you say. And I gather you are responding mostly to what I said to BKBK. In some points you  are restating what I did, and in others you are clarifying.

 

Bottom line: I do understand that you need to set the read timeout. I do understand that you don't want or need to change the ldap server. I do understand that you're getting a socketread hangup. That is VERY common whenever ANYthing in CF is talking to something outside of it (like a cfquery or cfhttp) and that thing hangs up.

 

2) So now, we wait to see first if your colleage has any luck with the jndi.properties. As for your noting that the CF admin (its "java and jvm" page) has a classpath with those other couple of folders, please don't misinterpret that. That is NOT the entire classpath. Indeed, if you go to the "settings summary" (next to the "java and jvm" page), and search for "class path" (with a space), you will see that there are two, with literally dozens of files and paths set in CF's class paths.

 

And while none alone says only \cfusion\lib; (or yourinstance\lib), you will see that there are again literally dozens of jar files named there which ARE in that folder. And while none refers to jndi, the point is that the jar that DOES load JNDI MAY well be in that folder and so it MAY look in that folder to find its properties file. I shared previously that the Java 11 jndi doc you ponited to DID indicate that it WOULD look in the classpath. That's why I have high hopes for it. (And you should only consider trying it elsewhere if that does NOT work. But I will say that the oosdk folders are about the open office sdk, and I can't see them having any connection to ldap.)

 

3) If that props file still does not work, we can dig into setting the env var. I will note that we CAN set those in the startup of CF. You don't say if you are running Windows or Linux. There are different ways (even in each), so let us know both what OS you use and how you (your admin) starts CF. I suspect it's as a service, but please confirm. But again I press this only if the properties file does not work.

 

4) Finally, now that you have share the stack trace, I note something quote interesting: it sure seems that this hangup is NOT while waiting for the ldap server to respond with your ldap query results. Don't get me wrong. I realize you are not GETTING those results. But what I mean is that if you look down that stack trace, the state it's in is:

 

coldfusion.tagext.net.LdapTag.closeConnections(LdapTag.java:870)

It's trying to CLOSE the connection. It seems like it thinks it GOT its result, and just wants to close the connection. I could be wrong, but I just want to note it.

 

And FWIW, above that it's also doing: sun.security.ssl.SSLSocketImpl.close. That means you were doing a secure connection (ssl or tls). I'm just curious if you can check something (again, if the jndi.properties doesn't work): can you tell the cfldap to NOT use an ssl/tls connection? I realize you may feel that you MUST. I just ask if you have tried it, just for this one failing cfldap call, and just for a single test call. If somehow that DID work, then the problem may well have to do more with that than the actual response from the server. Again, I could be barking up thr wrong tree. I just wanted to share it while I observed it.


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 01, 2021 Apr 01, 2021

Copy link to clipboard

Copied

Say, we forget for the moment about JNDI.properties and "com.sun.jndi.ldap.read.timeout". I wish to look at this afresh, from the very beginning. 

 

Could you please share your cfldap tag in full. Please use xxx to obfuscate any private information. For example, 

server="xxx"
username="xxx"
password="xxx"

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 01, 2021 Apr 01, 2021

Copy link to clipboard

Copied

Not to seem contrary, but George had indeed said the other day that his admin would be trying the props file in a new location, so I am indeed looking forward to if that helped. Or there was my suggestion about implementing the environment variables via the startup script (rather than via coding). Any word on either, George? 


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 04, 2021 Apr 04, 2021

Copy link to clipboard

Copied

@Charlie Arehart : Not to seem contrary, but George had indeed said the other day

 

No worries. We can follow different lines of enquiry simultaneously. 🙂

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Apr 05, 2021 Apr 05, 2021

Copy link to clipboard

Copied

All, sorry for the delay in responding. My server admin wasn't able to try until just a little bit ago.

 

We placed the jndi.properties file in {application.home}/lib/oosdk/lib and {application.home}/lib. In the JVM.config the oosdk path was in the java.class.path and the /lib was in the java.library path. After a reboot there was no change in behaviour.   I know no way to verify whether the jndi.properties file is being loaded or the value set, so the only check I can do is whether I'm still getting hung connections or not.

 

@Charlie Arehart"Finally, now that you have share the stack trace, I note something quote interesting: it sure seems that this hangup is NOT while waiting for the ldap server to respond with your ldap query results."
I noticed that to, but it happens immediately after the start tag, and there's several decode and read actions after that, so I just assumed it's part of the handshake.  I attempted to try and compare it with a non-hung stack trace, but haven't figured out how to do that in FusionReactor yet.

 

@BKBK"Could you please share your cfldap tag in full. Please use xxx to obfuscate any private information."

This is an example of the cfldap call from a test page I'm using:
cfldap( action="query"
,server="XXX"

,port=389
,scope="subtree"
,attributes="employeeID "
,filter="(&(objectClass=organizationalPerson)(sAMAccountName=XXX*))"
,maxrows="10"
,name="results"
,start="OU=XXX,OU=XXX,DC=XXX,DC=XXX,DC=XXX,DC=XXX"
,username="XXX"

,password="XXX"
,usetls="yes"
);

I had the timeout previously specified, but removed after one of your earlier suggestions.   If I add timeout with something really short like 250ms I can generate the occasional timeout error.  So the timeout value works, it just doesn't work with the ones that are randomly hanging.

 

If I remove the usetls, or set usetls to "No" or try port 636 I get a connection error. I can use port 3268 but they don't have a correct LDAP schema applied so several of the fields will return empty. I still received random hangs with port 3268.

 

There's no rhyme or reason to when it hangs, refreshing the same test page might work 50 times in a row and then randomly hang on the 51st.   The LDAP server admins say they see no issues on their end, and to their network people it appears that our CF server is failing to close the connection.   Which might explain why it looks to me like their server sends the close at the 900 second point when it hits the Max Connection Idle time.   Unfortunately there's no one outside our organization hitting this particular LDAP server so I can't compare with other users.   Even though no one else is using it the LDAP server admin isn't willing to change the configuration to set the max idle connection to a lower value.

 

@Charlie Arehart"Or there was my suggestion about implementing the environment variables via the startup script (rather than via coding)."
I'm open to trying it, just be aware that it'll probably take at least several days before I can get the server admin to give it a try.  CF is running on a Windows 2016 Server as a service. The Java versions is 11.0.10.  We're still on CF2018 Update 10, but should be going to update 11 by next week at the latest.

 

Please let me know if I failed to answer anyones question.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 05, 2021 Apr 05, 2021

Copy link to clipboard

Copied

George, I do look forward to hearing if the env var approach (via an updated start file) may help you.

 

And as for the properties approach, perhaps you lost track of it but I had said last week that you may be heading down the wrong path in putting the file in the oosdk folder (which has to do with openoffice). I have been suggesting from the beginning instead the cfusion/lib folder (or instancename/lib).

 

I appreciate that you may feel "they are both in the classpath, so it should not matter". And it may not, but I just think that as long as you will be getting someone to try to get the env var approach working, if it does NOT (or perhaps even trying before that), it could be useful to know if the properties file approach WOULD have worked. (Twiddling with startup scripts is notoriously risky and brittle. Still, I know your are likely just frustrated and desparate to try anything to get things to work.)

 

One last sanity check: I know you have confirmed twice that the jvm that CF is running is indeed 11.0.10 (the current latest Java 11 update). But while one may discern that from the CF Admin of your instance, I wuold just propose that a last resort is to instead run some simple code to output the JVM version--just in case something about connecting the dots via the Admin could be amiss. It's worth a shot. And I did a blog post just today on this (for a similar but different problem).  It's just one line of code, if you'd be willing to at least rule it out for us.

 

And if it DOES confirm again that you are 11.0.10, I look forward to hear what your next testing may show us. (Or perhaps others will have still different suggestions. And yep, BKBK, I do of course agree with your last comment.)


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

@Charlie Arehart I do look forward to hearing if the env var approach (via an updated start file) may help you.

I'm not understanding what you want me to try?   We already tried setting it int the JVM.config.   It's started as a Windows Service.

 

I have been suggesting from the beginning instead the cfusion/lib folder (or instancename/lib)

I did the {application.home}/lib folder in addition to the oosdk folder.    cfusion/lib was tried during the intial attempt.

 

I wuold just propose that a last resort is to instead run some simple code to output the JVM version.

That's how I've pulled the JVM version each time.   I even tried it again with your sample code.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

George, I realize this post has become a book. 🙂 And anyone could be forgiven for not being able to keep track of every point and nuance. 🙂

 

As for your first question here (about setting the env var in other than the jvm.config), I had said earlier:

 

"If that props file still does not work, we can dig into setting the env var. I will note that we CAN set those in the startup of CF. You don't say if you are running Windows or Linux. There are different ways (even in each), so let us know both what OS you use and how you (your admin) starts CF. I suspect it's as a service, but please confirm. But again I press this only if the properties file does not work."

 

As for your second point, I had said earlier (note the emphasis):

 

"You say also that you tried to set this in a jndi.proporties file, but you were stumped as to where to put the file. You said you "tried JAVA_HOME\lib and runtime\lib". Can you be more specific? Those can be translated into various locations (by you or by us reading your words), and there are indeed various lib folders within CF. 

 

More specifically, did you try it in the cfusion/lib folder, right under your main CF folder? Or if you are running an instance, then its sibling instancename/lib folder? Those are the folders where one would modify the various jars related to JDBC drivers that CF loads, for instance).

 

And then you restarted CF, and then did your test, and it didn't help?"

 

Finally, as for your last point, while you did indeed say multiple times THAT you were on Java 11.0.10, you never confirmed HOW you found that. I pressed more than once asking, such as if you may have been getting it from the CF Admin (and I commented how one could be misled by that, if not careful). So in this last/previous comment of mine, I simply pressed the point to say you COULD be sure with a check via code. If you're now saying you were doing that all the while, ok. We couldn't have known.

 

I'm sure all this back and forth is as frustrating for you as for others (and myself). This is a knotty problem that you have. And such problems can often have really odd explanations, and it may take a while to get to the solution. And sometimes in these cases we end up smacking our heads that we couldn't see the problem more readily--or we may discover it's a new and gnarly problem with an important resolution that many could learn from.

 

But I can say from participating in these kind of threads for now nearly 40 years that sometimes it really does pay to be clear and specific (which means more words), and indeed to challenge people to make sure they have done exactly what they say (because people can mean differing things by the same words). It's not meant to annoy anyone, but I realize that it can. (And it's ok if you're not saying you're annoyed. It's just that your last note here seemed to feel like you are saying things to me that you have already said, and as you can see, I don't quite agree.)


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

Charlie,

 

I apoligize if my responses came off as frustrated, I was just trying to keep them short and on point in an attempt to be clear. I really do appreciate yours and BKBK's assistance.

 

"More specifically, did you try it in the cfusion/lib folder, right under your main CF folder? Or if you are running an instance, then its sibling instancename/lib folder?"

The instancename/lib was done in the most recent attempt. I believe cfusion/lib was done previously, but I've gone back to my server admin and asked him to put the jndi.properties file there so we can try again. It might be days before he can restart the instance. As I mentioned this is shared environment and others are a little frustrated with the multiple restarts.

 

"I suspect it's as a service, but please confirm. But again I press this only if the properties file does not work."
CF is being started as a service on Windows Server 2016.  I know you mention waiting until after trying the jndi.properties file in the cfusion/lib folder, but if I know what change you want me to try I can get the server admin to try that immediately afterwards if it doesn't work.


"I simply pressed the point to say you COULD be sure with a check via code."
I get it, I've been looking at your posts for years and have seen where it turns out ColdFusion wasn't pointed at the Java the user thought it was.  Or they're looking at the wrong jvm.config file.  Me pointing out your code snippet was just an attempt to be specific on how I pulled it.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

@George Alsobrooks , Thanks for the information. Comprehensive.

 

The way I see it, your cfldap attributes are fine. I would bring back the timeout attribute. Set it to the number of milliseconds you want. 

 

As we continue looking into read-timeout, we should rule out other possibilities.

 

I am now leaning towards the idea that a possible cause of this problem might be the server. In particular, a flaw in the TLS implementation at the server might occasionally prevent clients from timing out.

 

Please find out what the Server's Operating System and TLS version are. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

@BKBKDo you happen to know how to check on a Windows 2016 server what version of TLS is being used for the LDAP call?   I could ask my server admin, but I suspect they're not going to know.   I found a couple web pages explaining how to check what versions are allowed, but that doesn't narrow down which one is being used by the LDAP call.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 06, 2021 Apr 06, 2021

Copy link to clipboard

Copied

@George Alsobrooks , the TLS settings are in the registry, I think. To be sure, google: windows server tls registry

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Apr 29, 2021 Apr 29, 2021

Copy link to clipboard

Copied

@Charlie Arehartand @BKBK 

Apoligies to both of you for the delay in responding. It took awhile to do the additional tests.  I also tried using GetPageContext() to get the context and set the property, but I suspect the LDAP call occurs under a different context, or I simply don't know how to set it that way.

 

I don't think I'm going to be able to do anything more with this. I've been tasked onto other projects and shouldn't be working on this issue anymore, but if either of you have any other thoughts I'll try and get those tested as soon as I can. I mostly wanted to post again to thank both of you and let you know that I appreciate all the help.

 

It looks like Adobe accepted my bug report CF-4211448 . It's currently marked as Bug Verified and To Fix, so who knows I may see a fix someday.

 

To answer the last questions I was asked:

"More specifically, did you try it in the cfusion/lib folder, right under your main CF folder? Or if you are running an instance, then its sibling instancename/lib folder?"

I had the server admin redo instancename/lib and cfusion/lib to make sure that both were tried, there was no change in behaviour.
Everything I've read online about the jndi.properties file said Java should be gathering it from a lot of places, so the fact none of the ones I've tried have worked leads me to think I'm just on the completely wrong path with trying to use a jndi.properites file to fix it.

 

"Please find out what the Server's Operating System and TLS version are."
It's Windows 2016 Server. My server admin responded back that it's TLSv1.2. I'd take that with a grain of salt. I supect without actually inspecting the LDAP connection it's hard to tell what's actually being used.

 

Thanks again to both of you and I really appreciate the support both of you give to the ColdFusion community.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 29, 2021 Apr 29, 2021

Copy link to clipboard

Copied

Thanks for the follow-up, the bug report, and especially the kind regards. Bummer that it remains unresolved (for now).

 

Yep, let's see what Adobe may do with that bug report. Great that they verified it, but as you say that doesn't guarantee if or when it will be fixed. I've added my vote, as I'm interested to hear how it turns out.

 

And sure, if any of us may have new thoughts on the matter, I suppose it would be best to offer them there (since Adobe don't tend to pay attention to anything said here, or perhaps not in the context of working on such a bug report). Then again, some won't see my suggestion here, so they may well still offer more thoughts here in the future. Perhaps you or we can suggest then if it's something they should add to the bug report as well. Always a challenge when the place a topic is discussed forks like that.


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines