CF11 Update 5, IIS 7 - Server Unavailable

[sdsinc_pmascari]

So, we're having a issue where some of our longer running processes are crashing with a 503 IIS error of Server Unavailable.  We have tried all the suggested settings of tuning the connectors using the "workers.properties" and "server.xml" files.  We've rebuilt all the connectors after the most recent updates.  I've even disabled IIS Rapid Fail.  But the issue continues to happen.

We have CF11 Update 5, Win 2008 R2, IIS 7.

The crash usually occurs at around 60 seconds of a page load.  All the timeout settings I can find I've set for 240 seconds or more yet we keep hitting this 60 second wall.

The only thing that looks to be a clue is in the ISAPI logs which seems to show the connector constantly restarting.  Also, we suspect some of our processes are taking extra long to complete due to this known bug.  I believe this is related to an issue Ben Nadel has also blogged about.

We've had huge processes in the past that took 4 to 5 minutes to complete without issue.  But that was on CF 8, 9, 10.  This is a relatively new install of CF11 and we need help.

What can we do to stop this connector from crashing!?  ISAPI log below:

[Tue May 05 18:35:44.157 2015] [3068:4660] [info] init_jk::jk_isapi_plugin.c (2976): Starting Jakarta/ISAPI/isapi_redirector/1.2.40

[Tue May 05 18:35:44.172 2015] [3068:4660] [info] init_jk::jk_isapi_plugin.c (3174): Jakarta/ISAPI/isapi_redirector/1.2.40 initialized

[Tue May 05 18:35:44.360 2015] [3068:4660] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 1 of max reuse connection 250 and total endpoint count 250

[Tue May 05 19:01:44.742 2015] [3068:2408] [info] TerminateFilter::jk_isapi_plugin.c (2661): Jakarta/ISAPI/isapi_redirector/1.2.40 stopping

[Wed May 06 08:57:13.747 2015] [4576:6016] [info] init_jk::jk_isapi_plugin.c (2976): Starting Jakarta/ISAPI/isapi_redirector/1.2.40

[Wed May 06 08:57:13.778 2015] [4576:6016] [info] init_jk::jk_isapi_plugin.c (3174): Jakarta/ISAPI/isapi_redirector/1.2.40 initialized

[Wed May 06 08:57:15.684 2015] [4576:6016] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 1 of max reuse connection 250 and total endpoint count 250

[Wed May 06 09:40:14.178 2015] [4576:2320] [info] TerminateFilter::jk_isapi_plugin.c (2661): Jakarta/ISAPI/isapi_redirector/1.2.40 stopping

[Wed May 06 10:01:46.473 2015] [4380:4188] [info] init_jk::jk_isapi_plugin.c (2976): Starting Jakarta/ISAPI/isapi_redirector/1.2.40

[Wed May 06 10:01:46.489 2015] [4380:4188] [info] init_jk::jk_isapi_plugin.c (3174): Jakarta/ISAPI/isapi_redirector/1.2.40 initialized

[Wed May 06 10:01:48.972 2015] [4380:4188] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 1 of max reuse connection 250 and total endpoint count 250

[Wed May 06 11:01:46.408 2015] [4380:4548] [info] TerminateFilter::jk_isapi_plugin.c (2661): Jakarta/ISAPI/isapi_redirector/1.2.40 stopping

[Wed May 06 11:07:57.320 2015] [5432:4892] [info] init_jk::jk_isapi_plugin.c (2976): Starting Jakarta/ISAPI/isapi_redirector/1.2.40

[Wed May 06 11:07:57.320 2015] [5432:4892] [info] init_jk::jk_isapi_plugin.c (3174): Jakarta/ISAPI/isapi_redirector/1.2.40 initialized

[Wed May 06 11:07:57.414 2015] [5432:4892] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 1 of max reuse connection 250 and total endpoint count 250

[Wed May 06 11:08:05.178 2015] [5432:5076] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 2 of max reuse connection 250 and total endpoint count 250

[Wed May 06 11:08:05.210 2015] [5432:4788] [info] ajp_process_callback::jk_ajp_common.c (2164): current reuse count is 3 of max reuse connection 250 and total endpoint count 250

[carl type3]

Hi,

I think your ISAPI logs are not an indicator of a problem.

You say the long process typically worked ok CF8 CF9 CF10 but not CF11. Do you know if CF8 CF9 CF10 JVM arguments were adjusted from default install settings? If so perhaps similar JVM settings can be applied to CF11. Some caution tho CF8 CF9 CF10 were likely using Java 6 or 7, CF11 could be using Java 8. Some JVM arguments differ in syntax with Java 8 versa older Java 6 and 7.

I guess details recorded in coldfusion-error or coldfusion-out logs would show a fail message - Server Unavailable. 

HTH, Carl.

[sdsinc_pmascari]

Thanks for the reply.  However, we believe this to most definitely a connector problem.  The reason being that whatever process ColdFusion was in the middle of, it continues to process after the crash.  For instance, if a file was being created, the crash will happen but we can monitor the progress of the CF scripts and see that the file still ends up where it's supposed to.  There are no Server Unavailable messages in the coldfusion-error nor coldfusion-out logs...none.

We scrounged around and found an old CF10 isapi.dll connector file and installed it.  So far, no more problems.

I have a suspicion that what we're experience is related to this posting about a bug in the connector causing application pool to stop. This latest CF update (5) was supposed to fix it but it doesn't appear so for us.

[carl type3]

Hi,

Lets hope the CF10 ISAPI connector on CF11 works for you.

Since you have altered the connector are you still getting ISAPI log "Jakarta/ISAPI/isapi_redirector/1.2.40 initialized" messages?

I see I did not write something very well. You will not find "server unavailable" in the CF "error" or "out" logs. Those are more likely to mention an error or warning leading up to the problem when you get the browser displaying server unavailable. One possible error message could include "Java heap space".

Regards, Carl.

[sdsinc_pmascari]

Wow!  Didn't see this one coming.  The problem ended up being our cloud provider's load balancer!  It had a timeout setting so as to disconnect long running processes.  This setting is not available on their normal dashboard.  On a hunch I wrote support and asked it this timeout setting existed and was told it did.  I asked for it to be increased for us and our crashes have stopped.  Totally explains how we were seeing nothing in the log files.

Apologies to all for wasting time thinking this was a connector issue.