connection error when updating to CF 2018 v11. Possible issue with wsconfig?

[dawalkertiger]

I have CF 2018 Enterprise, and was trying to update from v4 to v11. I have IIS running on a web server (different machine) and CF running on another machine (app server) (I think this is called "Distributed Mode"). v4 everything works as it should, but when I upgrade to v11, I see the following errors in isapi_redirect.txt file (I run wsconfig.ext on the webserver to establish the connection between the web server (IIS) and app server (CF). Here are the errors I am seeing in the isapi_redirect.txt

[Sat Aug 21 10:55:10.690 2021] [5916:5976] [info] jk_open_socket::jk_connect.c (816): connect to [CF App server IP address]:8018 failed (errno=61)
[Sat Aug 21 10:55:10.690 2021] [5916:5976] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1140): (cfusion) Failed opening socket to ([CF App server IP address]:8018) (errno=61)
[Sat Aug 21 10:55:10.690 2021] [5916:5976] [error] ajp_send_request::jk_ajp_common.c (1811): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Sat Aug 21 10:55:10.690 2021] [5916:5976] [info] ajp_service::jk_ajp_common.c (2982): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [info] jk_open_socket::jk_connect.c (816): connect to [CF App server IP address]:8018 failed (errno=61)
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1140): (cfusion) Failed opening socket to ([CF App server IP address]:8018) (errno=61)
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [error] ajp_send_request::jk_ajp_common.c (1811): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [info] ajp_service::jk_ajp_common.c (2982): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=2)
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [error] ajp_service::jk_ajp_common.c (3003): (cfusion) connecting to tomcat failed (rc=-3, errors=10, client_errors=0).
[Sat Aug 21 10:55:11.830 2021] [5916:5976] [error] HttpExtensionProc::jk_isapi_plugin.c (2734): service() failed with http error 503

Again, to fix, I simply revert back to v4.

CF seemed to be running fine on v11. The port 8018 was listening, and all the CF processes were running.

What am I missing

Because I couldn't find this forum initially, I posted this here https://coldfusion.adobe.com/2021/08/connection-error-updating-cf-2018-v11/#comment-46153

I did get an answer there saying that v8 update had a tomcat change that required server.xml to be manually updated with the ip of the cf server. Is this the answer? Or is there something else to do? It does seem this is a bug if we have to manually update configuration after the fact.

[RaviShankar Chagnur]

Hello,

You can add 127.0.0.1 in server.xml file, for more information you can refer to below document under section "ERROR 503"

https://helpx.adobe.com/coldfusion/kb/coldfusion-2018-update-8.html#troubleshoot

Thanks & Regards

Ravi

[Charlie Arehart]

While Ravi recommends editing the server.xml file to put in 127.0.0.1 as the address (on the AJP connector line, as discussed in that technote he links to), I suspect you will instead need to put in the IP address of the machine which is running IIS.

This is not a bug, but rather a "security enhancement" that was added (not by Adobe but by Tomcat) in early 2020, where they specifically locked down the Tomcat connector (which is running inside CF, as configured in that server.xml file) so that by default it could only be reached from a web server that was expected to be running on the same machine (so defaulting to 127.0.0.1). The "fix" in this case is to modify that file to name the actual IP address of your IIS server.

And this change was introduced by the  CF updates from Mar 2020 (CF2018 update 8, and CF2016 update 14), which incorporated the Tomcat connector update. And so now, anyone who jumps from any update level BEFORE those to any update level AFTER those would also hit the issue. (There is yet another, having to do with possible additional headers that your web server could send, which also could be rejected by default.)

FWIW, besides that Adobe technote about the update (which each subsequent update does point to at its top), I also did a coupel of blog posts at that time, a "briefer" one on the Adobe site, and an elaborated one on my own site. Sadly, I think Adobe has determined not to change the wsconfig tool to try to accomodate this problem (whether by trying to autodetect things, or by providing a means for you to identify what change might be needed).

Let us know if you get going.

[dawalkertiger]

Thanks everyone! I think I have a good path forward here. I hope to try this in the next few days if I can get the situation to reoccur in my test/dev environment (it's not set up exactly the same way as production). Otherwise, I may have to retry live. Either way, I'll report back.

Thanks again!

[dawalkertiger]

Charlie,

I finally found time to attempt this solution and it did not work. Here is the AJP connector from my server.xml and the error I was seeing in coldfusion-error log. Any thoughts?

nnn.nnn.nnn.nnn is IP address of the IIS server
mmmm is the port

<Connector packetSize="65535" address="nnn.nnn.nnn.nnn" port="mmmm" protocol="AJP/1.3" redirectPort="yyyy" tomcatAuthentication="false" maxThreads="500" connectionTimeout="60000"/>

INFO: Initializing ProtocolHandler ["ajp-nio-nnn.nnn.nnn.nnn-mmmm"]
Dec 13, 2021 7:12:59 PM org.apache.catalina.util.LifecycleBase handleSubClassException
SEVERE: Failed to initialize component [Connector[AJP/1.3-mmmm]]
org.apache.catalina.LifecycleException: Protocol handler initialization failed
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1056)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardService.initInternal(StandardService.java:561)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at org.apache.catalina.core.StandardServer.initInternal(StandardServer.java:1049)
at org.apache.catalina.util.LifecycleBase.init(LifecycleBase.java:136)
at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:954)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:564)
at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:100)
at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:185)
Caused by: java.net.BindException: Cannot assign requested address: bind
at java.base/sun.nio.ch.Net.bind0(Native Method)
at java.base/sun.nio.ch.Net.bind(Net.java:433)
at java.base/sun.nio.ch.Net.bind(Net.java:425)
at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:225)
at org.apache.tomcat.util.net.NioEndpoint.initServerSocket(NioEndpoint.java:272)
at org.apache.tomcat.util.net.NioEndpoint.bind(NioEndpoint.java:227)
at org.apache.tomcat.util.net.AbstractEndpoint.bindWithCleanup(AbstractEndpoint.java:1208)
at org.apache.tomcat.util.net.AbstractEndpoint.init(AbstractEndpoint.java:1221)
at org.apache.coyote.AbstractProtocol.init(AbstractProtocol.java:603)
at org.apache.catalina.connector.Connector.initInternal(Connector.java:1053)
... 12 more

[Charlie Arehart]

The error indicates "Cannot assign requested address: bind", which indicates that the port it's referring to is in use. Sadly, it doesn't name the port but I'd suspect whatever you chose for mmmm. Try a different port.

And if you stumble and don't want to wait for back and forth here, we could solce ALL of this quickly in a remote screenshare session, via my carehart.org/consulting. 

[BKBK]

@dawalkertiger , I would solve the problem as follows:

1.  (Assuming this hasn't been done yet) Download and install jdk-11.0.12_windows-x64_bin.exe from the ColdFusion Downloads site, and use it as the Java engine for ColdFusion 2018;
2. If you did 1., then restart ColdFusion and verify that everything is working as expected;
3. Upgrade to Update 8 (from the current Update 4) ;
4. Apply the post-installation steps for Update 8;
5.  Recreate the connector by running the Web Server Configuration Tool, C:\ColdFusion2018\cfusion\runtime\bin\wsconfig.exe;
6. Verify that everything is working as expected in CF2018 Update 8. Verify, in particular, that the isapi-redirect errors no longer occur;
7. Restart ColdFusion;
8.  Upgrade to Update 11 (from Update 8);
9.  (Optionally) Recreate the connector by running the Web Server Configuration Tool, C:\ColdFusion2018\cfusion\runtime\bin\wsconfig.exe;

[Charlie Arehart]

Bkbk, can you offer any thoughts on why you think those are the steps dawalkertiger needs to take, given both what he said, and Ravi, and me? It really seems to take none of that into account.

Specifically, what do your suggested steps have to do with his ip address problem? You're basically saying "start over", when there's nothing to suggest that one of those steps was missed. (And I don't see any connection to the Java version used.) 

Your step 4 is the key point, and it's what Ravi and I already said, though with more focus on the ip address aspects of that, specifically. 

Finally, there's no reason at all to do update 8 separately from update 11. The updates are cumulative, and as I noted, all the update technotes after 8 tell folks to apply its post install steps. That seems the crux of the problem here, and the rest of the steps you list seem unnecessary. 

I really am asking to advance the conversation, and so that we all might learn, not merely to be contradictory. My hope is that we all work together to help solve problems. Often your answers are great, but if I see something amiss I ask in order to help us all, just as I welcome such from anyone about my replies. 

[BKBK]

@Charlie, my suggestion has humbler intentions than the points you make. I do not think that those are the steps Dawalkertiger needs to take. They are the steps I would follow.

[Charlie Arehart]

Well, you surely mean the steps you would take if you had his problem (otherwise why propose them). And in that case, I'd ask you the same questions.

But if you want to leave it at this, that's fine. Let's indeed focus on solving the original problem. 

[BKBK]

Well, you surely mean the steps you would take if you had his problem

By @Charlie Arehart

Yes.