503 errors after ~3 months w/ CF10

[vertizonal]

Server is not responding about every other day now. The service states that it's running. Restarting fixes it temporarily.

My first thought was that this was a client variables issue, since the 90 day purge causes people a lot of problems. But I've confirmed that the codebase doesn't reference "clientmanagement". I moved client variable storage to a database anyway (nothing is getting populated there), and turned global updates off. It still hangs after doing this.

There's not much in the logs (there are no hs*.log files, and no exceptions in the regular CF logs), and so far Fusion Reactor has not pointed to any smoking guns. About the only thing I can find is in the isapi_redirect.log file (ColdFusion10/config/wsconfig/1/):

[Wed Jul 23 18:44:05.215 2014] [13144:10964] [info] ajp_process_callback::jk_ajp_common.c (2066): current reuse count is 118 of max reuse connection 250 and total endpoint count 500

[Wed Jul 23 18:44:05.217 2014] [13144:6856] [info] ajp_process_callback::jk_ajp_common.c (2066): current reuse count is 119 of max reuse connection 250 and total endpoint count 500

[Wed Jul 23 19:48:52.294 2014] [13144:6344] [info] jk_open_socket::jk_connect.c (626): connect to 127.0.0.1:8012 failed (errno=61)

[Wed Jul 23 19:48:52.295 2014] [13144:6344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1047): Failed opening socket to (127.0.0.1:8012) (errno=61)

[Wed Jul 23 19:48:52.299 2014] [13144:6344] [error] ajp_send_request::jk_ajp_common.c (1669): (cfusion) connecting to backend failed. Tomcat is probably not started or is listening on the wrong port (errno=61)

[Wed Jul 23 19:48:52.301 2014] [13144:6344] [info] ajp_service::jk_ajp_common.c (2692): (cfusion) sending request to tomcat failed (recoverable), because of error during request sending (attempt=1)

[Wed Jul 23 19:48:53.402 2014] [13144:6344] [info] jk_open_socket::jk_connect.c (626): connect to 127.0.0.1:8012 failed (errno=61)

[Wed Jul 23 19:48:53.406 2014] [13144:6344] [info] ajp_connect_to_endpoint::jk_ajp_common.c (1047): Failed opening socket to (127.0.0.1:8012) (errno=61)

This leads me to believe the problem may be either with the IIS Connector, or with Tomcat itself, but I'm not sure where to go from here, if that's the case. We ran all the CF10 updates through update 12, and didn't initially redo the IIS Connector through wsconfig.exe, and restart IIS, but did about a month or so ago.

We also adjusted the JVM settings a bit:

-XX:MaxPermSize=192m --> 256m

and set the min and max JVM heap size to 2048 (from 1024). We have 6GB of RAM on the dedicated server.

Here's the JVM memory usage from the hour of the crash (green dots at the bottom indicate the server restart)

The ramp up without any garbage collection is the same thing we saw the last time the server crashed as well. I have the Fusion Analytics logs from the latest crash, if anybody would like to see a specific chart.

Any suggestions would be greatly appreciated. Thanks.

[carl type3 · Accepted solution]

hi,

From isapi log errors would appear tomcat at the CF end is not responding to the iis tomcat connector. Likely you need to make some adjustments or tune values to server.xml in cfusion\runtime\conf.

What values? This CF11 blog entry may help since the details also apply to CF10.

http://blogs.coldfusion.com/post.cfm/coldfusion-11-iis-connector-tuning

Keep in mind when making adjustments to server.xml matching changes need to be applied to workers.properties in config\wsconfig\N.

You could also do some monitoring with free jdk tools like jconsole to help know what tuning values would suit the type of load best. Java and tomcat monitoring tho is time and effort so reply  on thread if you want to know more about how to do monitoring.

HTH, Carl.

[vertizonal]

Carl,

Thanks for the info. That's actually the last thing I adjusted yesterday - making the server.xml file match workers.properties, so I'm glad to see you suggesting doing the same. The Connector node didn't have the maxThreads nor connectionTimeout attribute, so maybe that's the issue. We also did a planned server restart last night, as I had read that this is sometimes the only way to get IIS Connector changes to stick.

One thing I did notice is that the Executor node values are different as well:

<Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="65" namePrefix="catalina-exec-"></Executor>

  Should maxThreads be the same as on Connector?

> reply  on thread if you want to know more about how to do monitoring.

  I'd definitely like to use JConsole for tuning, especially if my last changes don't help. Any advice you can give me about this would be greatly appreciated.

Thanks,

Dave

[Anit_Kumar]

Just, as a caution, make sure that the AJP entry in server.xml should look like

<Connector port="8012" protocol="AJP/1.3" redirectPort="8445" tomcatAuthentication="false" maxThreads="500" connectionTimeout ="60000"> </Connector>

vertizonal wrote:

<Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="65" namePrefix="catalina-exec-"></Executor>

  Should maxThreads be the same as on Connector?

Thanks,

Dave

Also, the maxThreads value is equivalent to the connection_pool_size in the worker.properties.

Regards,

Anit Kumar

[vertizonal]

Anit,

Thanks - the Connector node looks exactly like that.

I'm a little unclear - should I change the Executor node maxThreads to be 500 as well? If so, does this require a CF and/or IIS restart?

Dave

[Anit_Kumar]

No, changes in Executor node are not required.

Any changes made to {CF-Home}/config/wsconfig/{some no}/, including isapi_redirect.dll or worker.properties, would require an IIS restart. Any changes made to {CF-Home}/cfusion/runtime/conf/server.xml requires “ColdFusion 11 Application Server” service restart.

Regards,

Anit Kumar

[vertizonal]

Ok - thanks, Anit.

Hopefully the server will work correctly now. I'll follow up next week - hopefully I won't need to sooner.

Dave

[carl type3]

Hi Dave,

I discuss CF tomcat tuning here. Likely the demo will interest you most. The slide deck link is still available:

http://experts.adobeconnect.com/p8l51p4s9m4

HTH again, Carl.

[vertizonal]

The server has been stable now for about four days now, which is much longer than it has been before I synced up the server.xml and workers.properties files.

Thank you Carl and Anit for all your help. Thanks also for the screencast, Carl. I will look at further tuning the server based on your suggestions in the screencast, but I want to make sure the server is definitely stable first.