Connector errors after ColdFusion 2018 Update 5: Bad gateway, Tomcat is down or refused connection

[BKBK]

We have been getting frequent errors after installing Update 5 of ColdFusion 2018. The site serves web pages, but pressing F5 (to refresh the page) produces an Apache error page: 

The isapi_redirect.log file entry corresponding to the failed request is:

[Wed Oct 23 17:18:44.903 2019] [13452:8640] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1425): (t1stdrdrci) can't receive the response header message from tomcat, tomcat (127.0.0.1:8030) has forced a connection close for socket 1464
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] ajp_get_reply::jk_ajp_common.c (2402): (t1stdrdrci) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] ajp_service::jk_ajp_common.c (2975): (t1stdrdrci) sending request to tomcat failed (unrecoverable), (attempt=1)
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] HttpExtensionProc::jk_isapi_plugin.c (2739): service() failed with http error 502

We applied the patched isapi_redirect.dll that Adobe recently released. In vain.

I then reported a bug: https://tracker.adobe.com/#/view/CF-4205448 (Please follow this link and vote to have this fixed - Thnx!)

Our setup:

ColdFusion 2018 Enterprise Update 5

64 Bit Windows Server 2016 + IIS 

If you found a solution to this problem, could you please share it with us?

[Charlie Arehart]

BKBK, you say this is since applying update 5. Did you first try to revert to whatever update you were on before, to prove is this is indeed "the cause"? And if you may not want to do that, have you tried using whatever isapi_redirect.dll you had from BEFORE the update 5? Sadly, the CF wsconfig "update" feature does not save a backup (I have a feature request for that).

And in doing the manual update to the dll, did you restart IIS after that? 

Was it failing the same way after update 5 but before the new dll? Had you tried to upate the connector AS PER THE UPDATED CONNECTOR THAT CAME with update 5? Or did you perhaps skip that step? What update were you on before?

Finally, it's possible that your problem is NOT about the update at all. I assume the site was working fine before the update, right? Did you do ANYTHING else after applying the CF update? Did you update the connector (with the wsconfig, or not)? Did you change anything in IIS? Did you perhaps "remove" and re-add a connector, instead of using the "update" feature of the wsconfig tool?

Answers to any of the above could provide a clue as to what is amiss. I appreciate folks often just wish to "get the solution", but as you are finding this is a more challenging problems for you, and it may take a bit more digging to resolve. (I will assume you are not interested in a consulting session to try to address this, if the above does not help. But I will say to other readers who may find this that I can often help solve such things, and perhaps in just minutes when I can actually see what's amiss on your server.)

Hope something there helps.

[BKBK]

Hi Charlie,

Thanks for your feedback.

The issue does not occur when we revert to our previous version, ColdFusion 2018 Update 4.

We changed nothing in IIS. In doing the manual dll update, we did restart IIS and recycle the application pool.

We did indeed update as per the updated connector that came with update 5. In fact, we later tried 2 more patched isapi_redirect.dll which the Adobe ColdFusion Team was kind enough to send us. In one of the tests we updated the connector using wsconfig. (We didn't remove and re-add the connector). Still, no dice. All our attempts failed in the same way, with the same error messages.

There was one thing we did after applying Update 5. As a workaround for the issue https://tracker.adobe.com/#/view/CF-4205271, we added the following flag in JVM config

-Dcoldfusion.http.usepooling=false

Do you suggest we try the isapi_redirect.dll of Update 4 on the current Update 5 installation?

[Charlie Arehart]

Oh, ok. Good that you can recover by reverting. That was not clear (so I wasn't sure how desparate the situation was). Of course, I realize you may have a pressing/urgent desire to get to update 5, for its bug fixes.

And to be clear, is the DLL you are trying with u5 the one Adobe offers here: https://coldfusion.adobe.com/2019/10/error-accessing-coldfusion-administrator-using-connector-port/? There is a link for it in that blog post. If you have not tried it, it's worth a shot.

And then sure, if it does NOT work, then you could try the u4 connector with the u5 update in place. It has worked for some, and yet not for others. Odd. I just mean that if you did go forward to update 5 (and the updated one from Adobe still did NOT work), then you COULD try the older connector (from u4 or before).  If neither works, you could revert back to u4 and its connector dll.

Either way, if you feel confident that the problem is SOLELY in the update 5 and even that updated DLL they offer, this is where Adobe is saying (in their discussions of the issue) that they want to hear from folks, at cfinstal@adobe.com (one "L"). So yes, I am proposing that rather than (or in addition to) opening forum threads and bug tracker tickets, take their offer to reach out about this specific issue, because of course they TOO want to see people get to update 5, and they want to understand who still has problems with its updated connector.

I will say it's still possible there's SOMETHING in your environment to explain things, but I will leave it for them to sort that out as I am trusting your assertion that there is nothing else "changing" between it working with u4 and failing with u5. 🙂

[BKBK]

Thanks, Charlie.

We shall extract the isapi_redirect.dll from Update 4 (which worked fine), and test it on Update 5.

[ashwinia70450482]

Hi Charlie:

We have the same problem, in one of our cf servers, but restarting the service will fix it. It occurs off and on, so now way i can reproduce this. 

[Charlie Arehart]

Ashwinia, when you say you have the "same problem", do you mean exactly what BKBK originally reported, about "can't receive the response header message from tomcat, tomcat (127.0.0.1:8030) has forced a connection close for socket" (where the port will differ for your server)? 

If instead you mean you see the message, "Tomcat is down or refused connection", then beware that may be a totally "different problem", which may have totally different solutions than those discussed here. 

So as for solving your issue, you have not indicated what cf version you're on, what update level it is, whether you've updated your cf web server connector, whether you're using iis or apache, whether you have done any connector "tuning", etc. And are you sure there are no errors in cf itself, or hung requests, which may otherwise explain  the "Tomcat down" error that is resolved by a cf restart? (Of course, cf runs on tomcat, so the reference in the error to tomcat is indeed a reference to cf). 

If you want to get our help "solving" this with you, we need that info. And if you may want it solved more quickly than via back and forth here, I can help via consulting as well.

Let's hear more from you on all the above. 

[ashwinia70450482]

HTTP/1.1 502 Bad Gateway Server: Microsoft-IIS/10.0 Date: Tue, 29 Dec 2020 19:33:09 GMT Connection: close Content-Type:text/html

Bad Gateway!

There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.

 
  Tomcat/ISAPI/isapi_redirector/1.2.46
 This one Charlie

[ashwinia70450482]

I meant Isapi redirect error part in BKBK issue

[ashwinia70450482]

We are on cf 2018, IIS, current with updates. Let me know, if you need anything else from me

[Charlie Arehart]

We will not be able to solve your problem readily with only that info. You may have problems in cf (having nothing to do with the connector) , or you may have need of connector tuning. You may have failed to update the connector. These are things we can't easily diagnose remotely.

I know you just want the problem solved, and likely preferably here for free. I do offer hundreds of replies per year for free, but I just don't think that will work well for this. Perhaps others will throw out many things to try.

But my sense is that this will be best and most effectively resolved via a direct remote screenshare session. If you're interested, see the link in my signature (including my satisfaction guarantee). We might get to the bottom of things in less than a half hour. 

[ashwinia70450482]

Charlie:

I have updated the connector again, it fixed the problem. 

[Charlie Arehart]

Very good to hear. It can surprise many that it could be such a simple solution. It is lamentable that we have to do it separately from the cf update itself. And while the cf update technote tells us if a given update calls for it, so many never see that at the bottom  of the technotes (since cf11).

Glad it's resolved for you. All the better to enjoy the new year. 🙂 

[BKBK]

ashwinia70450482

That one caused us a headache for months. But we finally found a solution. 🙂

Charlie has asked you some vital questions. You have yet to answer them:

• the update level of your CF2018 installation
• whether or not you've updated your web server connector
• whether or not you have "tuned" the connector
• whether or not you have checked the ColdFusion and Tomcat logs to rule out issues in ColdFusion itself

[BKBK]

An update.

One part of our application runs on CFCs and CFM pages. The other is a REST API.

With the assistance of the ColdFusion Team at Adobe, we have discovered the following:

The 502 Bad Gateway occurs only in the REST API. We narrowed the root-cause down to the onRESTRequest event-handler in Application.cfc. The fact that OnRESTRequest is not single-threaded is the root-cause.

It turns out that, in our application, multiple REST requests sharing the same URI do occasionally have access to onRESTRequest. Some of the code in onRESTRequest involves flushing a servlet response outputstream to the client, as follows:

var out = response.getOutputStream();     
out.write(toBinary(toBase64(content)));       
out.flush();       
out.close();

As onRESTRequest is not single-threaded, two such responses sometimes create a race-condition. One of them is flushed to the client, as expected. But parts of the second response (for example, headers) may not be flushed or may be flushed in the wrong order. The result is a 502 Bad Gateway error.

Workaround:

Use a ColdFusion flush after the servlet flush.

var out = response.getOutputStream();     
out.write(toBinary(toBase64(content)));       
out.flush();       
out.close();

// Ensure ColdFusion flushes every response
cfflush();

[BKBK · Accepted solution]

In the end, we found an even neater alternative to the above solution, namely:

if (isResponseContentBinary) {
    // Stream binary content
    writeoutput(toString(binaryResponse));
    cfflush();
} else {
    // Return REST response
    restSetResponse(response);
    return response;
}