Skip to main content
BKBK
Community Expert
October 23, 2019
Answered

Connector errors after ColdFusion 2018 Update 5: Bad gateway, Tomcat is down or refused connection

  • October 23, 2019
  • 2 replies
  • 3110 views

We have been getting frequent errors after installing Update 5 of ColdFusion 2018. The site serves web pages, but pressing F5 (to refresh the page) produces an Apache error page: 

The isapi_redirect.log file entry corresponding to the failed request is:

 

[Wed Oct 23 17:18:44.903 2019] [13452:8640] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1425): (t1stdrdrci) can't receive the response header message from tomcat, tomcat (127.0.0.1:8030) has forced a connection close for socket 1464
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] ajp_get_reply::jk_ajp_common.c (2402): (t1stdrdrci) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] ajp_service::jk_ajp_common.c (2975): (t1stdrdrci) sending request to tomcat failed (unrecoverable), (attempt=1)
[Wed Oct 23 17:18:44.903 2019] [13452:8640] [error] HttpExtensionProc::jk_isapi_plugin.c (2739): service() failed with http error 502

 

We applied the patched isapi_redirect.dll that Adobe recently released. In vain.

I then reported a bug: https://tracker.adobe.com/#/view/CF-4205448 (Please follow this link and vote to have this fixed - Thnx!)

 

Our setup:

ColdFusion 2018 Enterprise Update 5

64 Bit Windows Server 2016 + IIS 

 

If you found a solution to this problem, could you please share it with us?

 

This topic has been closed for replies.
Correct answer BKBK

In the end, we found an even neater alternative to the above solution, namely:

if (isResponseContentBinary) {
    // Stream binary content
    writeoutput(toString(binaryResponse));
    cfflush();
} else {
    // Return REST response
    restSetResponse(response);
    return response;
}

2 replies

BKBK
BKBKAuthor
Community Expert
May 31, 2020

An update.

One part of our application runs on CFCs and CFM pages. The other is a REST API.

With the assistance of the ColdFusion Team at Adobe, we have discovered the following:

 

The 502 Bad Gateway occurs only in the REST API. We narrowed the root-cause down to the onRESTRequest event-handler in Application.cfc. The fact that OnRESTRequest is not single-threaded is the root-cause.

 

It turns out that, in our application, multiple REST requests sharing the same URI do occasionally have access to onRESTRequest. Some of the code in onRESTRequest involves flushing a servlet response outputstream to the client, as follows:

 

var out = response.getOutputStream();     
out.write(toBinary(toBase64(content)));       
out.flush();       
out.close();

 

As onRESTRequest is not single-threaded, two such responses sometimes create a race-condition. One of them is flushed to the client, as expected. But parts of the second response (for example, headers) may not be flushed or may be flushed in the wrong order. The result is a 502 Bad Gateway error.

 

Workaround:

Use a ColdFusion flush after the servlet flush.

 

var out = response.getOutputStream();     
out.write(toBinary(toBase64(content)));       
out.flush();       
out.close();

// Ensure ColdFusion flushes every response
cfflush();

 

 

BKBK
BKBKAuthorCorrect answer
Community Expert
December 30, 2020

In the end, we found an even neater alternative to the above solution, namely:

if (isResponseContentBinary) {
    // Stream binary content
    writeoutput(toString(binaryResponse));
    cfflush();
} else {
    // Return REST response
    restSetResponse(response);
    return response;
}
Charlie Arehart
Community Expert
October 23, 2019

BKBK, you say this is since applying update 5. Did you first try to revert to whatever update you were on before, to prove is this is indeed "the cause"? And if you may not want to do that, have you tried using whatever isapi_redirect.dll you had from BEFORE the update 5? Sadly, the CF wsconfig "update" feature does not save a backup (I have a feature request for that).

 

And in doing the manual update to the dll, did you restart IIS after that? 

 

Was it failing the same way after update 5 but before the new dll? Had you tried to upate the connector AS PER THE UPDATED CONNECTOR THAT CAME with update 5? Or did you perhaps skip that step? What update were you on before?

 

Finally, it's possible that your problem is NOT about the update at all. I assume the site was working fine before the update, right? Did you do ANYTHING else after applying the CF update? Did you update the connector (with the wsconfig, or not)? Did you change anything in IIS? Did you perhaps "remove" and re-add a connector, instead of using the "update" feature of the wsconfig tool?

 

Answers to any of the above could provide a clue as to what is amiss. I appreciate folks often just wish to "get the solution", but as you are finding this is a more challenging problems for you, and it may take a bit more digging to resolve. (I will assume you are not interested in a consulting session to try to address this, if the above does not help. But I will say to other readers who may find this that I can often help solve such things, and perhaps in just minutes when I can actually see what's amiss on your server.)

 

Hope something there helps.

/Charlie (troubleshooter, carehart. org)
ashwinia70450482
Participating Frequently
December 29, 2020

Hi Charlie:

We have the same problem, in one of our cf servers, but restarting the service will fix it. It occurs off and on, so now way i can reproduce this. 

ashwinia70450482
Participating Frequently
December 29, 2020

Ashwinia, when you say you have the "same problem", do you mean exactly what BKBK originally reported, about "can't receive the response header message from tomcat, tomcat (127.0.0.1:8030) has forced a connection close for socket" (where the port will differ for your server)? 

 

If instead you mean you see the message, "Tomcat is down or refused connection", then beware that may be a totally "different problem", which may have totally different solutions than those discussed here. 

 

So as for solving your issue, you have not indicated what cf version you're on, what update level it is, whether you've updated your cf web server connector, whether you're using iis or apache, whether you have done any connector "tuning", etc. And are you sure there are no errors in cf itself, or hung requests, which may otherwise explain  the "Tomcat down" error that is resolved by a cf restart? (Of course, cf runs on tomcat, so the reference in the error to tomcat is indeed a reference to cf). 

 

If you want to get our help "solving" this with you, we need that info. And if you may want it solved more quickly than via back and forth here, I can help via consulting as well.

 

Let's hear more from you on all the above. 


HTTP/1.1 502 Bad Gateway Server: Microsoft-IIS/10.0 Date: Tue, 29 Dec 2020 19:33:09 GMT Connection: close Content-Type:text/html

Bad Gateway!

There is a problem with the page you are looking for, and it cannot be displayed. When the Web server (while acting as a gateway or proxy) contacted the upstream content server, it received an invalid response from the content server.


 
  Tomcat/ISAPI/isapi_redirector/1.2.46
 This one Charlie