Coldfusion 10 update 18 cluster instance restarts in hung state while under load

[GuitsBoy]

So this problem has been driving me completely bonkers.

CF10 update 18, RedHat 6.7 64 bit.  Java 1.7.093

We noticed a similar problem with coldfusion 10 update 14 last year, and rolled back to update 13.  Recently we pushed out update 18, which fixed some issues, but did not fix CF restarting into a hung or dead state.

I have gone as far as uninstalling CF10 completely, reinstalling a fresh CF10, doing the mandatory update, and then doing update 18.  Creating two fresh new instances, a new cluster, and configuring apache connector to use said cluster.  Its about as out of the box as I can make it.

With this physical box quiesced in my hardware load balancer (not taking requests) I can happily restart the two CF instances all day long without much problem (maybe the occasional hang, but rare).   The trouble starts when I put the box back into production and allow it to start taking requests, and then restart a CF instance.   At least one CF instance will always come up in a hung or dead state.  I cannot get to tomcat under port 8501, and mod_jk shows the worker as down.  The coldfusion_out log shows "PM Information [localhost-startStop-2] - ColdFusion stopped" as the last entry.  Running a ps aux shows that the instance is indeed running.  Issuing a kill -3 <pid> does not log anything to std err or to coldfusion-out, or anywhere else for that matter.  When I try to stop or restart the instance again, I get the following message, and again, the instance starts in a hung state

[root]# /opt/coldfusion10/web4cf1/bin/coldfusion restart               

Restarting ColdFusion 10 server instance named web4cf1 ...

Stopping ColdFusion 10 server instance named web4cf1, please wait

Mar 22, 2016 2:24:09 PM com.adobe.coldfusion.launcher.Launcher stopServer

SEVERE: Shutdown Port 8008is not active. Stop the server only after it is started.

ColdFusion 10 server instance named web4cf1 has been stopped

Starting ColdFusion 10 server instance named web4cf1 ...

The ColdFusion 10 server instance named web4cf1 is starting up and will be available shortly.

nohup: appending output to `nohup.out'

======================================================================

ColdFusion 10 server instance named web4cf1 has been started.

ColdFusion 10 will write logs to /opt/coldfusion10/web4cf1/logs/coldfusion-out.log

======================================================================

I have tried various timeouts and settings in the server.xml files as well as workers.properties with no luck.   Here are my current files:

worker.list=web4cluster,web4cf1,web4cf2

worker.web4cluster.type=lb

worker.web4cluster.balance_workers=web4cf1,web4cf2

worker.web4cluster.sticky_session=true

worker.web4cf1.type=ajp13

worker.web4cf1.host=localhost

worker.web4cf1.port=8041

worker.web4cf1.connect_timeout=250

worker.web4cf1.max_reuse_connections=250

worker.web4cf1.connection_pool_size = 400

worker.web4cf1.connection_pool_minsize= 200

worker.web4cf1.connection_pool_timeout = 600

worker.web4cf1.lbfactor=1

worker.web4cf1.route=web4cf1

worker.web4cf2.type=ajp13

worker.web4cf2.host=localhost

worker.web4cf2.port=8042

worker.web4cf2.connect_timeout=250

worker.web4cf2.max_reuse_connections=250

worker.web4cf2.connection_pool_size = 400

worker.web4cf2.connection_pool_minsize= 200

worker.web4cf2.connection_pool_timeout = 600

worker.web4cf2.lbfactor=1

worker.web4cf2.route=web4cf2

<Server port="8008" shutdown="SHUTDOWN">

  <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">

  </Listener>

  <Listener className="org.apache.catalina.core.JasperListener">

  </Listener>

  <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">

  </Listener>

  <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">

  </Listener>

  <GlobalNamingResources>

    <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">

    </Resource>

  </GlobalNamingResources>

  <Service name="Catalina">

    <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">

    </Executor>

    <Connector port="8501" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="2000" redirectPort="8446" executor="tomcatThreadPool" maxThreads="50">

    </Connector>

    <Connector port="8041" protocol="AJP/1.3" redirectPort="8446" tomcatAuthentication="false">

    </Connector>

    <Engine jvmRoute="web4cf1" name="Catalina" defaultHost="localhost">

      <Realm className="org.apache.catalina.realm.LockOutRealm">

        <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">

        </Realm>

      </Realm>

      <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">

        <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">

        </Valve>

      </Host>

      <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelSendOptions="8">

        <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">

        </Manager>

        <Channel className="org.apache.catalina.tribes.group.GroupChannel">

          <Membership port="45564" dropTime="3000" address="228.0.240.104" className="org.apache.catalina.tribes.membership.McastService" frequency="500">

          </Membership>

          <Receiver port="4001" autoBind="100" address="auto" selectorTimeout="2000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver">

          </Receiver>

          <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">

            <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender">

            </Transport>

          </Sender>

          <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector">

          </Interceptor>

          <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor">

          </Interceptor>

        </Channel>

        <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter="">

        </Valve>

        <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve">

        </Valve>

        <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener">

        </ClusterListener>

        <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener">

        </ClusterListener>

      </Cluster>

    </Engine>

  </Service>

</Server>

Now if I stop apache, or if I remove the load by removing the physical box from the hardware load balancer, the coldfusion instance will happily restart (after the port 8008 not active error).

If I start the coldfusion instance manually from the command line, I do get some error messages when it fails:

[root]# /opt/coldfusion10/jre/bin/java -classpath /opt/coldfusion10/web4cf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/web4cf2/bin/cf-bootstrap.jar:/opt/coldfusion10/web4cf2/lib/oosdk/lib:/opt/coldfusion10/web4cf2/lib/oosdk/lib/*:/opt/coldfusion10/web4cf2/lib/oosdk/classes:/opt/coldfusion10/web4cf2/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/web4cf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/web4cf2 -Dcoldfusion.libPath=/opt/coldfusion10/web4cf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/web4cf2/lib/updates,/opt/coldfusion10/web4cf2/lib,/opt/coldfusion10/web4cf2/lib/axis2,/opt/coldfusion10/web4cf2/gateway/lib/,/opt/coldfusion10/web4cf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/web4cf2/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

Mar 22, 2016 3:45:36 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /opt/coldfusion10/jre/lib/amd64/server:/opt/coldfusion10/jre/lib/amd64:/opt/coldfusion10/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Mar 22, 2016 3:45:36 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-bio-8502"]

Mar 22, 2016 3:45:36 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["ajp-bio-8042"]

Mar 22, 2016 3:45:36 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service Catalina

Mar 22, 2016 3:45:36 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

Mar 22, 2016 3:45:36 PM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

INFO: Cluster is about to start

Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.transport.ReceiverBase bind

INFO: Receiver Server Socket bound to:/10.10.240.104:4002

Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

INFO: Setting cluster mcast soTimeout to 500

Mar 22, 2016 3:45:36 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:4

Mar 22, 2016 3:45:36 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 104}:4001,{10, 10, 240, 104},4001, alive=3337299, securePort=-1, UDP Port=-1, id={-65 -3 47 75 8 -4 74 2 -123 -42 -41 -22 84 88 60 82 }, payload={}, command={}, domain={}, ]

Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:4

Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 1000 milliseconds to establish cluster membership, start level:8

Mar 22, 2016 3:45:37 PM org.apache.catalina.tribes.io.BufferPool getBufferPool

INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:38 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:8

Mar 22, 2016 3:45:38 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:39 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.ha.session.ClusterSessionListener messageReceived

WARNING: Context manager doesn't exist:localhost#

Mar 22, 2016 3:45:40 PM org.apache.catalina.startup.TldConfig execute

INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Mar 22, 2016 3:45:41 PM org.apache.catalina.session.StandardSession tellNew

SEVERE: Session event listener threw exception

java.lang.NullPointerException

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:154)

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.java:69)

        at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

        at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

        at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

        at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

        at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

        at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)

        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Mar 22, 2016 3:45:41 PM org.apache.catalina.session.StandardSession tellNew

SEVERE: Session event listener threw exception

java.lang.NullPointerException

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:154)

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.java:69)

        at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

        at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

        at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

        at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

        at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

        at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)

        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)

        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

        at java.lang.Thread.run(Thread.java:662)

Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager startInternal

INFO: Register manager localhost# to cluster element Engine with name Catalina

Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager startInternal

INFO: Starting clustering manager at localhost#

Mar 22, 2016 3:45:41 PM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions

INFO: Manager [localhost#], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 104}:4001,{10, 10, 240, 104},4001, alive=3341800, securePort=-1, UDP Port=-1, id={-65 -3 47 75 8 -4 74 2 -123 -42 -41 -22 84 88 60 82 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds.

Mar 22, 2016 3:45:47 PM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions

INFO: Manager [localhost#]; session state send at 3/22/16 3:45 PM received in 5,614 ms.

Does anyone have any suggestions on what I could try?

[carl type3]

Hi,

I don't really know plus is not my particular environment but case it helps putting some idea's out for your consideration.

Some of my Windows CF clusters had to be on Amazon EC2. Amazon does not support tomcat clustering using multicast traffic so those are setup using tomcat static membership. Since I had a good experience using static membership method I prefer to use that even in non Amazon situations. So perhaps you might like to alter your cluster method.

One other that might be worth a try. Put your cluster settings in HOST section of server.xml. I don't know why but sometimes I have seen folks cluster problems solve but moving that. So server.xml would end like </Cluster></Host></Engine></Service></Server>

I know that last option trends against common knowledge or documentation but this is a forum, it is just a passing idea to a poster who has had no responses yet, please don't flame me.

HTH, Carl.

[GuitsBoy]

Hi Carl -

Thanks for the response.  Anyone willing to lend a helping hand or few ideas is certainly undeserving of being flamed no matter how against convention it may be, IMO.

I started to try static clusters, but didnt get it to work in the hour or so I tried, so I really didnt give it a solid effort yet.  I also tried the blocking Bio instead of Nio listeners/responders, as well as changing to backup manager but no dice there.  Perhaps tomorrow Ill try static members again.  Ill also rearrange server.xml to see if that gets us anywhere.

I was able to reproduce this error on a brand new fresh install of RHEL6.7 on a spare desktop.  Updated to CF 10 update 18, and created a new cluster as I always had.  I was able  to generate load using jmeter to hit a page that makes a large cfhttp call about 10 times a second.  This was enough load to get an instance to lockup on restart.  It restarts fine without the load but, once its working, it chokes on sending/receiving sessions.  I have increased all my timeouts up to 120 seconds, and still no good under any moderate load.

I found some additional information by calling java directly from the shell to run CF.  This way standard error is spit out onto the screen, and I can see where its choking.  Ill post my latest info tomorrow when I get back to work.

Anyway, Thanks again for the suggestions.

-T

[GuitsBoy]

When I start CF manually via command line, here's what I see.

This is instance 1:

Instance 1

[root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-bootstrap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*:/opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib,/opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start Mar 24, 2016 3:30:18 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Mar 24, 2016 3:30:18 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8503"] Mar 24, 2016 3:30:18 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-8015"] Mar 24, 2016 3:30:19 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Mar 24, 2016 3:30:19 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.64 Mar 24, 2016 3:30:19 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal INFO: Cluster is about to start Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.transport.ReceiverBase bind INFO: Receiver Server Socket bound to:/10.10.2.84:4001 Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket INFO: Setting cluster mcast soTimeout to 5000 Mar 24, 2016 3:30:19 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4 Mar 24, 2016 3:30:29 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Done sleeping, membership established, start level:4 Mar 24, 2016 3:30:29 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8 Mar 24, 2016 3:30:39 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Done sleeping, membership established, start level:8 Mar 24, 2016 3:30:41 AM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Register manager localhost# to cluster element Engine with name Catalina Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Starting clustering manager at localhost# Mar 24, 2016 3:30:41 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [localhost#]: skipping state transfer. No members active in cluster group. Mar 24, 2016 3:30:42 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services Mar 24, 2016 3:30:42 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01 Mar 24, 2016 3:30:48 AM org.apache.catalina.core.ApplicationContext log INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646 Mar 24, 2016 3:30:52 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal INFO: JvmRouteBinderValve started Mar 24, 2016 3:30:52 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8503"] Mar 24, 2016 3:30:52 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-8015"] Mar 24, 2016 3:30:52 AM com.adobe.coldfusion.launcher.Launcher run INFO: Server startup in 35420 ms Mar 24, 2016 3:31:19 AM org.apache.catalina.tribes.io.BufferPool getBufferPool INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=10027, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ] Mar 24, 2016 3:32:09 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=20035, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:09 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=20035, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:09 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)         at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)         at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357)         at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)         at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)         at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)         at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)         at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:497)         at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:487)         at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:463)         at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:450)         at org.apache.catalina.connector.Request.doGetSession(Request.java:3029)         at org.apache.catalina.connector.Request.getSession(Request.java:2386)         at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:897)         at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:974)         at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1067)         at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:361)         at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)         at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)         at coldfusion.filter.PathFilter.invoke(PathFilter.java:112)         at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)         at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)         at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)         at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)         at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)         at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)         at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)         at coldfusion.CfmServlet.service(CfmServlet.java:219)         at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)         at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)         at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)         at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)         at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)         at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)         at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)         at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)         at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:170)         at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)         at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:956)         at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)         at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:218)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:333)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=80037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:44 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=85037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:44 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=85037, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:44 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:51 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:51 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:51 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:52 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 3:32:52 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=95038, securePort=-1, UDP Port=-1, id={98 87 72 -113 -116 -55 75 62 -95 -36 -59 -26 -97 121 -46 72 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 3:32:52 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(30000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:843)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:815)         at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:539)         at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:524)         at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:506)         at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:419)         at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:343)         at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:437)         at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)         at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)         at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)         at java.lang.Thread.run(Thread.java:745)

Instance 2

[root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf2/bin/cf-bootstrap.jar:/opt/coldfusion10/newcf2/lib/oosdk/lib:/opt/coldfusion10/newcf2/lib/oosdk/lib/*:/opt/coldfusion10/newcf2/lib/oosdk/classes:/opt/coldfusion10/newcf2/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf2 -Dcoldfusion.libPath=/opt/coldfusion10/newcf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf2/lib/updates,/opt/coldfusion10/newcf2/lib,/opt/coldfusion10/newcf2/lib/axis2,/opt/coldfusion10/newcf2/gateway/lib/,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start Mar 24, 2016 3:31:15 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Mar 24, 2016 3:31:15 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8504"] Mar 24, 2016 3:31:15 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-8016"] Mar 24, 2016 3:31:16 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Mar 24, 2016 3:31:16 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.64 Mar 24, 2016 3:31:16 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal INFO: Cluster is about to start Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.transport.ReceiverBase bind INFO: Receiver Server Socket bound to:/10.10.2.84:4002 Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket INFO: Setting cluster mcast soTimeout to 5000 Mar 24, 2016 3:31:16 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4 Mar 24, 2016 3:31:19 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4001,{10, 10, 2, 84},4001, alive=60017, securePort=-1, UDP Port=-1, id={66 70 35 114 47 105 70 -54 -66 113 -45 -14 -100 109 -50 -63 }, payload={}, command={}, domain={}, ] Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Done sleeping, membership established, start level:4 Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8 Mar 24, 2016 3:31:26 AM org.apache.catalina.tribes.io.BufferPool getBufferPool INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:26 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:27 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:28 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:29 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:30 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:31 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:32 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:33 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:34 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:35 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers INFO: Done sleeping, membership established, start level:8 Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:36 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:37 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:38 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived WARNING: Context manager doesn't exist:localhost# Mar 24, 2016 3:31:39 AM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Mar 24, 2016 3:31:39 AM org.apache.catalina.session.StandardSession tellNew SEVERE: Session event listener threw exception java.lang.NullPointerException         at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:154)         at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.java:69)         at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)         at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)         at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)         at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)         at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)         at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)         at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)         at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)         at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745) Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Register manager localhost# to cluster element Engine with name Catalina Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Starting clustering manager at localhost# Mar 24, 2016 3:31:39 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [localhost#], requesting session state from org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4001,{10, 10, 2, 84},4001, alive=80018, securePort=-1, UDP Port=-1, id={66 70 35 114 47 105 70 -54 -66 113 -45 -14 -100 109 -50 -63 }, payload={}, command={}, domain={}, ]. This operation will timeout if no session state has been received within 60 seconds. Mar 24, 2016 3:32:09 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions INFO: Manager [localhost#]; session state send at 3/24/16 3:31 AM received in 30,167 ms.

[GuitsBoy]

I'm running into similar issues with static clusters as well, but it does seem more likely to continue starting up when encountering an error:

Instance 1

[root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-bootstrap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*:/opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib,/opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.IntrospectionUtils setProperty WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember uniqueId={1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}) java.lang.reflect.InvocationTargetException         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)         at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)         at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:182)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1344)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)         at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)         at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)         at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)         at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161) Caused by: java.lang.NullPointerException         at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)         at org.apache.catalina.tribes.membership.MemberImpl.setUniqueId(MemberImpl.java:616)         at org.apache.catalina.tribes.membership.StaticMember.setUniqueId(StaticMember.java:74)         ... 27 more Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'uniqueId' to '{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}' did not find a matching property. Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.IntrospectionUtils setProperty WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember domain=static-cluster) java.lang.reflect.InvocationTargetException         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)         at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)         at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:182)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1344)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)         at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)         at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)         at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)         at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161) Caused by: java.lang.NullPointerException         at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)         at org.apache.catalina.tribes.membership.MemberImpl.setDomain(MemberImpl.java:636)         at org.apache.catalina.tribes.membership.StaticMember.setDomain(StaticMember.java:65)         ... 27 more Mar 24, 2016 6:02:36 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'domain' to 'static-cluster' did not find a matching property. Mar 24, 2016 6:02:36 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Mar 24, 2016 6:02:36 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8503"] Mar 24, 2016 6:02:36 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-8015"] Mar 24, 2016 6:02:36 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Mar 24, 2016 6:02:36 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.64 Mar 24, 2016 6:02:36 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal INFO: Cluster is about to start Mar 24, 2016 6:02:36 AM org.apache.catalina.tribes.transport.ReceiverBase bind INFO: Receiver Server Socket bound to:/10.10.2.84:4001 Mar 24, 2016 6:02:38 AM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Register manager  to cluster element Host with name localhost Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Starting clustering manager at Mar 24, 2016 6:02:38 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager []: skipping state transfer. No members active in cluster group. Mar 24, 2016 6:02:40 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services Mar 24, 2016 6:02:40 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01 Mar 24, 2016 6:02:45 AM org.apache.catalina.core.ApplicationContext log INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646 Mar 24, 2016 6:02:49 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal INFO: JvmRouteBinderValve started Mar 24, 2016 6:02:49 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8503"] Mar 24, 2016 6:02:49 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-8015"] Mar 24, 2016 6:02:49 AM com.adobe.coldfusion.launcher.Launcher run INFO: Server startup in 15105 ms Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.io.BufferPool getBufferPool INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4002,10.10.2.84,4002, alive=0, securePort=-1, UDP Port=-1, id={1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ] Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck INFO: Suspect member, confirmed alive.[org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4002,10.10.2.84,4002, alive=0, securePort=-1, UDP Port=-1, id={1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 0 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]] Mar 24, 2016 6:03:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=1458813786146, securePort=-1, UDP Port=-1, id={-41 123 58 104 -22 -27 72 81 -103 -22 29 91 -58 -82 -98 23 }, payload={}, command={}, domain={}, ]] message. Will verify. Mar 24, 2016 6:03:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4002,{10, 10, 2, 84},4002, alive=1458813786146, securePort=-1, UDP Port=-1, id={-41 123 58 104 -22 -27 72 81 -103 -22 29 91 -58 -82 -98 23 }, payload={}, command={}, domain={}, ]] Mar 24, 2016 6:03:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send SEVERE: Unable to send message through cluster sender. org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;         at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)         at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)         at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)         at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)         at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)         at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)         at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357)         at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)         at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)         at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)         at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)         at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)         at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)         at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)         at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)         at java.lang.Thread.run(Thread.java:745)

Instance 2

[root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf2/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf2/bin/cf-bootstrap.jar:/opt/coldfusion10/newcf2/lib/oosdk/lib:/opt/coldfusion10/newcf2/lib/oosdk/lib/*:/opt/coldfusion10/newcf2/lib/oosdk/classes:/opt/coldfusion10/newcf2/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf2 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf2 -Dcoldfusion.libPath=/opt/coldfusion10/newcf2/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf2/lib/updates,/opt/coldfusion10/newcf2/lib,/opt/coldfusion10/newcf2/lib/axis2,/opt/coldfusion10/newcf2/gateway/lib/,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf2/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.IntrospectionUtils setProperty WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember uniqueId={0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}) java.lang.reflect.InvocationTargetException         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)         at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)         at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:182)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1344)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)         at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)         at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)         at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)         at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161) Caused by: java.lang.NullPointerException         at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)         at org.apache.catalina.tribes.membership.MemberImpl.setUniqueId(MemberImpl.java:616)         at org.apache.catalina.tribes.membership.StaticMember.setUniqueId(StaticMember.java:74)         ... 27 more Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'uniqueId' to '{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}' did not find a matching property. Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.IntrospectionUtils setProperty WARNING: IntrospectionUtils: InvocationTargetException for class org.apache.catalina.tribes.membership.StaticMember domain=static-cluster) java.lang.reflect.InvocationTargetException         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:303)         at org.apache.tomcat.util.IntrospectionUtils.setProperty(IntrospectionUtils.java:282)         at org.apache.tomcat.util.digester.SetPropertiesRule.begin(SetPropertiesRule.java:215)         at org.apache.tomcat.util.digester.Digester.startElement(Digester.java:1282)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.startElement(AbstractSAXParser.java:509)         at com.sun.org.apache.xerces.internal.parsers.AbstractXMLDocumentParser.emptyElement(AbstractXMLDocumentParser.java:182)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanStartElement(XMLDocumentFragmentScannerImpl.java:1344)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:2787)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:606)         at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:510)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:848)         at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:777)         at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)         at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)         at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:642)         at org.apache.tomcat.util.digester.Digester.parse(Digester.java:1555)         at com.adobe.coldfusion.launcher.Launcher.run(Launcher.java:200)         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)         at java.lang.reflect.Method.invoke(Method.java:606)         at com.adobe.coldfusion.bootstrap.Bootstrap.init(Bootstrap.java:86)         at com.adobe.coldfusion.bootstrap.Bootstrap.main(Bootstrap.java:161) Caused by: java.lang.NullPointerException         at org.apache.catalina.tribes.membership.MemberImpl.getData(MemberImpl.java:250)         at org.apache.catalina.tribes.membership.MemberImpl.setDomain(MemberImpl.java:636)         at org.apache.catalina.tribes.membership.StaticMember.setDomain(StaticMember.java:65)         ... 27 more Mar 24, 2016 6:03:05 AM org.apache.tomcat.util.digester.SetPropertiesRule begin WARNING: [SetPropertiesRule]{Server/Service/Engine/Host/Cluster/Channel/Interceptor/Member} Setting property 'domain' to 'static-cluster' did not find a matching property. Mar 24, 2016 6:03:05 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib Mar 24, 2016 6:03:05 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["http-bio-8504"] Mar 24, 2016 6:03:05 AM org.apache.coyote.AbstractProtocol init INFO: Initializing ProtocolHandler ["ajp-bio-8016"] Mar 24, 2016 6:03:06 AM org.apache.catalina.core.StandardService startInternal INFO: Starting service Catalina Mar 24, 2016 6:03:06 AM org.apache.catalina.core.StandardEngine startInternal INFO: Starting Servlet Engine: Apache Tomcat/7.0.64 Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal INFO: Cluster is about to start Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.transport.ReceiverBase bind INFO: Receiver Server Socket bound to:/10.10.2.84:4002 Mar 24, 2016 6:03:06 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded INFO: Replication member added:org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4001,10.10.2.84,4001, alive=0, securePort=-1, UDP Port=-1, id={0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ] Mar 24, 2016 6:03:06 AM org.apache.catalina.tribes.io.BufferPool getBufferPool INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl Mar 24, 2016 6:03:08 AM org.apache.catalina.startup.TldConfig execute INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time. Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Register manager  to cluster element Host with name localhost Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager startInternal INFO: Starting clustering manager at Mar 24, 2016 6:03:08 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions INFO: Manager [], requesting session state from org.apache.catalina.tribes.membership.StaticMember[tcp://10.10.2.84:4001,10.10.2.84,4001, alive=0, securePort=-1, UDP Port=-1, id={0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 }, payload={}, command={}, domain={115 116 97 116 105 99 45 99 108 ...(14)}, ]. This operation will timeout if no session state has been received within 60 seconds. Mar 24, 2016 6:03:13 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions INFO: Manager []; session state send at 3/24/16 6:03 AM received in 5,118 ms. Mar 24, 2016 6:03:14 AM org.apache.catalina.tribes.transport.nio.NioReplicationTask sendAck WARNING: Unable to send ACK back through channel, channel disconnected?: null Mar 24, 2016 6:03:14 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services Mar 24, 2016 6:03:14 AM org.apache.catalina.core.ApplicationContext log INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01 Mar 24, 2016 6:03:19 AM org.apache.catalina.core.ApplicationContext log INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646 Mar 24, 2016 6:03:24 AM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal INFO: JvmRouteBinderValve started Mar 24, 2016 6:03:24 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["http-bio-8504"] Mar 24, 2016 6:03:24 AM org.apache.coyote.AbstractProtocol start INFO: Starting ProtocolHandler ["ajp-bio-8016"] Mar 24, 2016 6:03:24 AM com.adobe.coldfusion.launcher.Launcher run INFO: Server startup in 20150 ms

I cant seem to figure out how to get rid of that error about the static member's uniqueID.  The syntax looks good, I dont get it.

Header 1
<Server port="8010" shutdown="SHUTDOWN">   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">   </Listener>   <Listener className="org.apache.catalina.core.JasperListener">   </Listener>   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">   </Listener>   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">   </Listener>   <GlobalNamingResources>     <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">     </Resource>   </GlobalNamingResources>   <Service name="Catalina">     <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">     </Executor>     <Connector port="8503" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="20000" redirectPort="8448" executor="tomcatThreadPool" maxThreads="50">     </Connector>     <Connector port="8015" protocol="AJP/1.3" redirectPort="8448" tomcatAuthentication="false">     </Connector>     <Engine jvmRoute="newcf1" name="Catalina" defaultHost="localhost">       <Realm className="org.apache.catalina.realm.LockOutRealm">         <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">         </Realm>       </Realm>       <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">         <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">         </Valve>       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelStartOptions="3" channelSendOptions="6">         <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">         </Manager>         <Channel className="org.apache.catalina.tribes.group.GroupChannel">           <!--<Membership port="45564" dropTime="10000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="5000">           </Membership>-->           <Receiver port="4001" autoBind="100" address="10.10.2.84" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver" timeout="30000"></Receiver>           <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">             <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"></Transport>           </Sender>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"></Interceptor>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"></Interceptor>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor"> <!--        <LocalMember port="4001" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}" domain="static-cluster" host="10.10.2.84"/>-->             <Member      port="4002" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}" domain="static-cluster" host="10.10.2.84"/>           </Interceptor>         </Channel>         <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""></Valve>         <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"></Valve>         <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"></ClusterListener>         <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"></ClusterListener>       </Cluster>       </Host>     </Engine>   </Service> </Server>
<Server port="8011" shutdown="SHUTDOWN">   <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on">   </Listener>   <Listener className="org.apache.catalina.core.JasperListener">   </Listener>   <Listener className="org.apache.catalina.core.JreMemoryLeakPreventionListener">   </Listener>   <Listener className="org.apache.catalina.mbeans.GlobalResourcesLifecycleListener">   </Listener>   <GlobalNamingResources>     <Resource description="User database that can be updated and saved" name="UserDatabase" pathname="conf/tomcat-users.xml" factory="org.apache.catalina.users.MemoryUserDatabaseFactory" type="org.apache.catalina.UserDatabase" auth="Container">     </Resource>   </GlobalNamingResources>   <Service name="Catalina">     <Executor name="tomcatThreadPool" minSpareThreads="4" maxThreads="150" namePrefix="catalina-exec-">     </Executor>     <Connector port="8504" protocol="org.apache.coyote.http11.Http11Protocol" connectionTimeout="20000" redirectPort="8449" executor="tomcatThreadPool" maxThreads="50">     </Connector>     <Connector port="8016" protocol="AJP/1.3" redirectPort="8449" tomcatAuthentication="false">     </Connector>     <Engine jvmRoute="newcf2" name="Catalina" defaultHost="localhost">       <Realm className="org.apache.catalina.realm.LockOutRealm">         <Realm className="org.apache.catalina.realm.UserDatabaseRealm" resourceName="UserDatabase">         </Realm>       </Realm>       <Host name="localhost" autoDeploy="false" unpackWARs="true" appBase="webapps">         <Valve pattern="%h %l %u %t &quot;%r&quot; %s %b" directory="logs" prefix="localhost_access_log." className="org.apache.catalina.valves.AccessLogValve" suffix=".txt" resolveHosts="false">         </Valve>       <Cluster className="org.apache.catalina.ha.tcp.SimpleTcpCluster" channelStartOptions="3" channelSendOptions="8">         <Manager notifyListenersOnReplication="true" expireSessionsOnShutdown="false" className="org.apache.catalina.ha.session.DeltaManager">         </Manager>         <Channel className="org.apache.catalina.tribes.group.GroupChannel">           <!--<Membership port="45564" dropTime="10000" address="228.0.0.4" className="org.apache.catalina.tribes.membership.McastService" frequency="5000">           </Membership>-->           <Receiver port="4002" autoBind="100" address="10.10.2.84" selectorTimeout="5000" maxThreads="6" className="org.apache.catalina.tribes.transport.nio.NioReceiver" timeout="30000"></Receiver>           <Sender className="org.apache.catalina.tribes.transport.ReplicationTransmitter">             <Transport className="org.apache.catalina.tribes.transport.nio.PooledParallelSender"></Transport>           </Sender>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.TcpFailureDetector"></Interceptor>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.MessageDispatch15Interceptor"></Interceptor>           <Interceptor className="org.apache.catalina.tribes.group.interceptors.StaticMembershipInterceptor">     <!--        <LocalMember port="4002" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,0}" domain="static-cluster" host="10.10.2.84"/>-->             <Member      port="4001" className="org.apache.catalina.tribes.membership.StaticMember" securePort="-1" uniqueId="{0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15}" domain="static-cluster" host="10.10.2.84"/>           </Interceptor>         </Channel>         <Valve className="org.apache.catalina.ha.tcp.ReplicationValve" filter=""></Valve>         <Valve className="org.apache.catalina.ha.session.JvmRouteBinderValve"></Valve>         <ClusterListener className="org.apache.catalina.ha.session.JvmRouteSessionIDBinderListener"></ClusterListener>         <ClusterListener className="org.apache.catalina.ha.session.ClusterSessionListener"></ClusterListener>       </Cluster>       </Host>     </Engine>   </Service> </Server>

[GuitsBoy]

Well, I spent all day playing around with static clusters, and unfortunately I'm in exactly the same place as with the multicast cluster.  I even stripped down the config and put Carl's syntax in there verbatim, save for IP addresses.  Still craps out under load.   Looks like somehow its just not communicating when a node recovers (ie. restarted).

Its still choking on these two errors primarily:

Mar 24, 2016 10:15:58 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send

SEVERE: Unable to send message through cluster sender.

org.apache.catalina.tribes.ChannelException: Operation has timed out(3000 ms.).; Faulty members:tcp://{10, 10, 2, 84}:4002;

        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:109)

        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:54)

        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:78)

        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:77)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)

        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:77)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)

        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:77)

        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:224)

        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:182)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:837)

        at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1429)

        at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1357)

        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1183)

        at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:944)

        at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:91)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:936)

        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:917)

        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:278)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.interceptors.TcpPingInterceptor.messageReceived(TcpPingInterceptor.java:170)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:117)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:82)

        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:252)

        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:287)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:210)

        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:99)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

Mar 24, 2016 10:15:58 AM org.apache.catalina.session.StandardSession tellNew

SEVERE: Session event listener threw exception

java.lang.NullPointerException

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.getListener(HttpFlexSessionBootstrap.java:154)

        at coldfusion.bootstrap.HttpFlexSessionBootstrap.sessionCreated(HttpFlexSessionBootstrap.java:69)

        at org.apache.catalina.session.StandardSession.tellNew(StandardSession.java:422)

        at org.apache.catalina.session.StandardSession.setId(StandardSession.java:394)

        at org.apache.catalina.ha.session.DeltaSession.setId(DeltaSession.java:275)

        at org.apache.catalina.ha.session.DeltaManager.handleSESSION_CREATED(DeltaManager.java:1317)

        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1195)

        at org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions(DeltaManager.java:803)

        at org.apache.catalina.ha.session.DeltaManager.startInternal(DeltaManager.java:756)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

        at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5526)

        at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150)

        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1574)

        at org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1564)

        at java.util.concurrent.FutureTask.run(FutureTask.java:262)

        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)

        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)

        at java.lang.Thread.run(Thread.java:745)

[GuitsBoy]

After playing with static clustering a few more days, I still ran headlong into a wall with this.

I decided to go back to multicast, and try BackupManager instead of DeltaManager, and it now seems to be restarting correctly, as long as my timout was high enough for the 15+ second AbstractMapReplication init.  Overall it takes about 65 seconds for server startup compared to 15 sec for static deltaManager and 30 sec for multicast deltaManager.  But the important part is that it eventually does start the server while under load, and session failover seems to work as expected.

Unfortunately using backupManager is not my preferred solution, but it may be the only option until I can figure out why DeltaManager is broken in our environment.  I would still love to hear if anyone has any suggestions as to why backup manager is working but delta manager is broken.

Thanks,

-Tony

[root@wp1 ~]# /etc/alternatives/jre_1.7.0/bin/java -classpath /opt/coldfusion10/newcf1/runtime/bin/tomcat-juli.jar:/opt/coldfusion10/newcf1/bin/cf-bootstrap.jar:/opt/coldfusion10/newcf1/lib/oosdk/lib:/opt/coldfusion10/newcf1/lib/oosdk/lib/*:/opt/coldfusion10/newcf1/lib/oosdk/classes:/opt/coldfusion10/newcf1/lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx2048m -XX:MaxPermSize=256m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home=/opt/coldfusion10/newcf1 -Djava.security.egd=file:/dev/./urandom -Dcoldfusion.rootDir=/opt/coldfusion10/newcf1 -Dcoldfusion.libPath=/opt/coldfusion10/newcf1/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dcoldfusion.classPath=/opt/coldfusion10/newcf1/lib/updates,/opt/coldfusion10/newcf1/lib,/opt/coldfusion10/newcf1/lib/axis2,/opt/coldfusion10/newcf1/gateway/lib/,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/flex/jars,/opt/coldfusion10/newcf1/wwwroot/WEB-INF/cfform/jars,/var/www/html/topaz/SigPlus2_29.jar,/var/www/html/comm.jar com.adobe.coldfusion.bootstrap.Bootstrap -start

Mar 28, 2016 2:17:34 PM org.apache.catalina.core.AprLifecycleListener lifecycleEvent

INFO: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib

Mar 28, 2016 2:17:34 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["http-bio-8503"]

Mar 28, 2016 2:17:34 PM org.apache.coyote.AbstractProtocol init

INFO: Initializing ProtocolHandler ["ajp-bio-8015"]

Mar 28, 2016 2:17:35 PM org.apache.catalina.core.StandardService startInternal

INFO: Starting service Catalina

Mar 28, 2016 2:17:35 PM org.apache.catalina.core.StandardEngine startInternal

INFO: Starting Servlet Engine: Apache Tomcat/7.0.64

Mar 28, 2016 2:17:35 PM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal

INFO: Cluster is about to start

Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.transport.ReceiverBase bind

INFO: Receiver Server Socket bound to:/10.10.2.84:4101

Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket

INFO: Setting cluster mcast soTimeout to 5000

Mar 28, 2016 2:17:35 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:4

Mar 28, 2016 2:17:37 PM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded

INFO: Replication member added:org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4202,{10, 10, 2, 84},4202, alive=655064, securePort=-1, UDP Port=-1, id={-18 29 64 7 95 -64 78 -124 -71 122 -116 113 -74 -10 31 -66 }, payload={}, command={}, domain={}, ]

Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:4

Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Sleeping for 10000 milliseconds to establish cluster membership, start level:8

Mar 28, 2016 2:17:45 PM org.apache.catalina.tribes.io.BufferPool getBufferPool

INFO: Created a buffer pool with max size:104857600 bytes of type:org.apache.catalina.tribes.io.BufferPool15Impl

Mar 28, 2016 2:17:55 PM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers

INFO: Done sleeping, membership established, start level:8

Mar 28, 2016 2:17:58 PM org.apache.catalina.startup.TldConfig execute

INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Mar 28, 2016 2:17:58 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap init

INFO: Initializing AbstractReplicatedMap with context name:-map

Mar 28, 2016 2:18:10 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap memberDisappeared

INFO: Member[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 2, 84}:4202,{10, 10, 2, 84},4202, alive=676062, securePort=-1, UDP Port=-1, id={-18 29 64 7 95 -64 78 -124 -71 122 -116 113 -74 -10 31 -66 }, payload={}, command={}, domain={}, ]] disappeared. Related map entries will be relocated to the new node.

Mar 28, 2016 2:18:10 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap memberDisappeared

INFO: Relocation of map entries was complete in 20 ms.

Mar 28, 2016 2:18:13 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap broadcast

WARNING: broadcast received 0 replies, probably a timeout.

Mar 28, 2016 2:18:13 PM org.apache.catalina.tribes.tipis.AbstractReplicatedMap init

INFO: AbstractReplicatedMap[-map] initialization was completed in 15300 ms.

Mar 28, 2016 2:18:17 PM org.apache.catalina.core.ApplicationContext log

INFO: ColdFusionStartUpServlet: ColdFusion: Starting application services

Mar 28, 2016 2:18:17 PM org.apache.catalina.core.ApplicationContext log

INFO: ColdFusionStartUpServlet: ColdFusion: VM version = 24.95-b01

Mar 28, 2016 2:18:30 PM org.apache.catalina.core.ApplicationContext log

INFO: CFMxmlServlet: Macromedia Flex Build: 87315.134646

Mar 28, 2016 2:18:37 PM org.apache.catalina.ha.session.JvmRouteBinderValve startInternal

INFO: JvmRouteBinderValve started

Mar 28, 2016 2:18:37 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["http-bio-8503"]

Mar 28, 2016 2:18:37 PM org.apache.coyote.AbstractProtocol start

INFO: Starting ProtocolHandler ["ajp-bio-8015"]

Mar 28, 2016 2:18:37 PM com.adobe.coldfusion.launcher.Launcher run

INFO: Server startup in 64767 ms