Instance in cluster will not restart

[chris_gulla]

CF11 w/update 6 cluster on Windows Server 2012 R2.  Secure profile.  The cluster includes two instances, one on a remote VM, and one a "home" instance (on same VM as IIS website and cfusion instance in which I create the cluster), and I think I set it up more or less in a standard way.  Everything works fine after I set it up.  Restarting the remote instance works fine.  Restarting the "home" instance--even after no changes to anything--results in the instance hanging during startup and may not be shut down except via Task Manager.  Any help from cluster experts out there would be greatly appreciated.  I'm happy to provide additional setup or diagnostic info.

Setup details

Nothing unusual about the configuration, except maybe that the home directory for the website is in an alternate directory per lockdown recommendations.  I also add maxThreads and connectionTimeout attributes to the Connector node for AJP/1.3 in server.xml.  When registering the remote instance I specify https.  I create the cluster to use sticky sessions and no session replication.  Restart both after creating the cluster.  Run the connector in the cfusion\runtime\bin folder and select advanced options to set the connection pool timeout to 60 seconds.  Works fine after that, as I get responses from both instances when requesting the website from two different browsers, as expected.  If I restart the remote instance, also no problems.

Details about restarting the "home" instance

Shutdown results in a few errors but the instance shuts down and I can attempt a restart.  (Two types of errors: many thrown by org.apache.catalina.loader.WebAppClassLoader, checkThreadLocalMapForLeaks, something about dates and creating a ThreadLocal with key of type coldfusion.util.dateUtils, failed to remove it when the web application was stopped.  The remote instance did this also when shutting down but it started ok.  The home instance also threw an error Illegal access: this web application instance has been stopped already.  Could not load java.util.concurrent.ScheduledFuture.  But it stops anyway.)

When I attempt to restart, the instance always hangs after this is logged to coldfusion-error.log:

Nov 17, 2015 10:46:51 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

INFO: Setting cluster mcast soTimeout to 500

Nov 17, 2015 10:46:51 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4

Nov 17, 2015 10:46:52 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:4

Nov 17, 2015 10:46:52 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:8

Nov 17, 2015 10:46:53 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:8

Nothing gets logged to any CF logs, the isapi_redirect.log, or Windows event logs during startup.

After this I cannot browse to the cfusion Instance Manager nor can I shut down using a command line.  I have to kill the process in Task Manager.

Thanks in advance for any help you can provide!

Chris

[carl type3]

Hi Chris,

I have not used the cfusion instance as part of a CF cluster. Perhaps you would like to try that on the home IIS server and the remote CF instance leave cfusion out of the configured instances and cluster.

I find leaving cfusion for creating instances, joining instances to make a cluster and managing the webserver connector WSCONFIG tends to work well.

HTH, Carl.

[chris_gulla]

Carl,

Thanks for your response but it looks like I misled-- sorry.  The instance I call my "home" instance ("DevHome" to be exact) is different than cfusion.  I call it "home" because it is on the same web server as my IIS website and the cfusion instance that I use to create my cluster.  I think I have to use the cfusion instance for that because it's the only CF Administrator that has Enterprise Manager on it.  The cluster includes only DevHome and my registered remote instance (DevRemote), not cfusion.  I hope this makes more sense now.

But, I had been using the wsconfig within the cfusion\runtime\bin folder because I figured that since cfusion runs the enterprise manager it should run the connector also.  Just to try it out, I used the wsconfig in DevHome\runtime\bin instead, which seemed to work fine.  Bad idea!  I made a request from my local machine that resulted in a perfectly fine response from DevHome, but a later request hung (from another browser to force the cluster to make DevRemote serve the page).  Not only that, but it crashed my web server!  The thing did a hard reboot.

Not sure if that is a clue to what my underlying problem is, or if that's actually what you were suggesting.  I'm guessing it was not what you were thinking though, so I will hold off on details.  But if this behavior is unexpected and you want logs there were a few things logged in the event logs that I would be happy to share.  Apart from that any further thoughts would be welcome.

Thanks,

Chris

[chris_gulla]

Carl,

More information for you.  I think the server rebooted as a result of the cfusion instance being unresponsive too many times in a row when I tried to start it up after commenting out the <Manager> tag as Rahul suggested.  So forget about the connector causing something that made the server reboot.

Either way, after the server came back I successfully started the DevHome instance from Instance Manager, and the cluster served pages from DevHome and DevRemote.  But again, shutting down the DevHome instance in Instance Manager and restarting it does not work.  It shuts down but hangs on startup at the same place I originally noted.  Now I know that restarting the server works, and is probably easier than rebuilding the cluster, but I'd rather not have to restart the server when all I need to do is restart the instance.

Thanks again for your thoughts and if you have any other ideas please pass them along.

Chris

[carl type3]

Hi Chris,

OK I understand that better now two instances that are clustered and not made up of either local or remote cfusion instance.

Agree with Rahul comment out the manager path in context.xml.

Currently I got not much more to offer other than something that I do not think will be related. If your CF servers are Amazon EC2 the multicast tomcat CF cluster method does not work due to deny on multicast traffic. I suppose your network between the clustered CF instances is not denying multicast so I feel I would be well wide of a solution to suggest setup the CF tomcat cluster using static members method. Perhaps if no other idea's come forward you might like to try that but wait for now until some of the other excellent posters have an offering.

Regards, Carl.

[chris_gulla]

Carl,

I'm not on AWS but did see that thread about not using multicast.  I like the idea so I tried it out but could not get it working.  At some point I may return to that and create another post asking for more help.  Thanks again for sharing your thoughts about this issue.

Chris

[rahul_u]

Hi Chris,

Please make sure you have "Manager pathname" commented in context.xml  (ColdFusion11\cfusion\runtime\conf)

<!--

    <Manager pathname="" />

    -->

Thanks,

Rahul

[chris_gulla]

Rahul,

Thanks for your suggestion but the cfusion instance does not start up when that line is commented out in its context.xml.  The windows system logs an error "The system cannot find the file specified."  I tried that a few times. Please also see my previous comment about which instances are in my cluster--this may change your recommendation.

Thanks,

Chris