Clustered CF Instances Hang, Difficult to Restart

Report · Jul 26, 2021

Hello all. Seems We have run into another issue with our newly deployed ColdFusion 2021 servers. We experienced a similar error back with CF8, and earlier update levels of CF10. It seems the issue was resolved in later updates of CF10, but now with CF2021, this issue seems to be back again.

We have three physical bare metal servers, each running four clustered instances of coldfusion. All of these are behind a fortinet load balancer. While the CF instances on each box are nammed the same (cfusion1 - cfusion4) each box's CF cluster is on a different port to eliminate multicast confusion between machines. We have changed channelSendOptions to 6 in the server.xml files in order to reduce the number of "Session Already Invalidated" error messages in the coldfusion-error.log files.

While we dont have much problem restarting instances when the server has been removed from the load balancer, we do see difficulty restarting instances even under moderate load. The CF instance will appear to start fine form the command line, however the instance never starts taking traffic, and the CFIDE/administrator for that instance will not load. Upon trying to stop the hung instance, we get an error:

[root@Node1 ~]# /opt/ColdFusion2021/cfusion1/bin/coldfusion start
Starting ColdFusion 2021 server ...
======================================================================
ColdFusion 2021 server has been started.
ColdFusion 2021 will write logs to /opt/ColdFusion2021/cfusion1/bin/../logs/coldfusion-out.log
======================================================================

[root@Node1 ~]# /opt/ColdFusion2021/cfusion1/bin/coldfusion stop
Stopping ColdFusion 2021 server, please wait
Jul 22, 2021 10:37:43 PM com.adobe.coldfusion.launcher.Launcher stopServer
SEVERE: Shutdown Port 8007is not active. Stop the server only after it is started.
ColdFusion 2021 server has been stopped

[root@Node1 ~]# /opt/ColdFusion2021/cfusion1/bin/coldfusion start
Starting ColdFusion 2021 server ...
======================================================================
ColdFusion 2021 server has been started.
ColdFusion 2021 will write logs to /opt/ColdFusion2021/cfusion1/bin/../logs/coldfusion-out.log
======================================================================

There doesn't appear to be an useful information in the coldfusion-error.log, nor the logs of it's peers.

Do I need to make any adjustments to the tomcat cluster timeouts perhaps? Amd I missing some other type of best practice when clustering CF instances? Any suggestions on how to troubleshoot this further?

Thanks for any advice,
-Tony

Report · Dec 22, 2021

Hello BKBK, Thank you again for the reply.

Unless my eyes are deceiving me, I only see one active instance of Manager in the context.xml, the first one is commented out. Again, this should be stock, as it ships from Adobe.

Yes, all instance of /wwwroot/WEB-INF/web.xml contain <distributable />

Channel Send Options of 6 did not seem to help anything, however it did not hurt anything either. It was decidedto keep it since we do not have terribly high throughput, and the reliability is more important than performance. We can try setting it back, however the problem will still persist, going by previous results.

Thanks again,
-Tony

Report · Dec 22, 2021

Unless my eyes are deceiving me, I only see one active instance of Manager in the context.xml, the first one is commented out.

By @GuitsBoy

I meant you should delete the one that has been commented out. Avoid repetition: the element should either be active or commented out. D.R.Y. 🙂

Yes, all instance of /wwwroot/WEB-INF/web.xml contain <distributable />

That's good.

Channel Send Options of 6 did not seem to help anything, however it did not hurt anything either. It was decidedto keep it since we do not have terribly high throughput, and the reliability is more important than performance. We can try setting it back, however the problem will still persist, going by previous results.

OK. Still worth a try. After all, some water has since passed under the bridge.

Report · Dec 22, 2021

I should also state that the channel send options of 6 eliminated the repeated "session already invalidates" messages that would fill our log files. Those are seemingly harmless, but certainly make reading the log file more challenging.

Report · Dec 22, 2021

And I will admit that when I saw this in the technote for update 2, I wondered what Adobe may have meant about "change the multicast port". Change it to what? How to know if it should be changed to some port that IS or IS NOT already in use. But I wanted to point this out for you in seeking "any suggestions".

By @Charlie Arehart

I think Adobe means you should change the the port number to something else. Say, from the current

<Membership address="228.0.0.1" port="45501"...>

to

<Membership address="228.0.0.1" port="45561"...>

Possible reasons for doing so:

avoid contention with some other process that requires port 45501
45501 may be obstructed in some way

In any case, @GuitsBoy , before you change the port number, ensure that the new port is free.

For example, on my Win 10, I can verify whether port 45561 is free by opening the Command Prompt as Administrator, then running the DOS command

netstat -ano | findstr "45561"

Talking of ports, there might be yet another consideration: the dynamic port range. Most servers now have the default range 49152 - 65535.

You can see your default TCP range by running the DOS command

netsh int ipv4 show dynamicport tcp

It might help if you expanded your range to include the port numbers in your Tomcat settings. For example, the command to expand the default TCP range to 10000 - 65535 is

netsh int ipv4 set dynamicport tcp start=10000 num=55536

Just a thought.

But beware! Network settings often require steady, expert hands. So consult your network guy or gal for this.

Report · Dec 22, 2021

Thanks again for your thoughts. Unfortunately, I happen to be he networking guy or gal.

We are running RHEL8 OS, not windows, however the equivalent netstat command shows the only ports in the 455xx range are indeed the ports assigned to tomcat.

I can also confirm that the ephermal port range covers the ports in question, 32768-60999.

Since this is talking to localhost, these ports should work even with the software firewall in place. However for troubleshooting, both the local firewall, as well as selinux have been disabled to avoid any blocking there. The servers are on their own private subnet, behind hardware loadbalancers, which are in turn behind a firewall, providing reasonable protection. Once things are working correctly, we would bring the software firewall back up with the appropriate ports passed though. Unfortunately it does not look like CF plays nicely with SELinux.

Report · Dec 22, 2021

Unfortunately, I happen to be he networking guy or gal.

By @GuitsBoy

Fortunately, I'd say. Fewer cooks in the kitchen. 🙂

We are running RHEL8 OS, not windows, however the equivalent netstat command shows the only ports in the 455xx range are indeed the ports assigned to tomcat.

I can also confirm that the ephermal port range covers the ports in question, 32768-60999.

Sounds good.

I can also confirm that the ephermal port range covers the ports in question, 32768-60999.

Since this is talking to localhost, these ports should work even with the software firewall in place. However for troubleshooting, both the local firewall, as well as selinux have been disabled to avoid any blocking there. The servers are on their own private subnet, behind hardware loadbalancers, which are in turn behind a firewall, providing reasonable protection. Once things are working correctly, we would bring the software firewall back up with the appropriate ports passed though. Unfortunately it does not look like CF plays nicely with SELinux.

Hmm, that rings a bell. I had actually been thinking: Tomcat network configuration, restricted or blocked access, that sort of thing. I wondered, for example: what would happen if you moved the <manager> element from context.xml to server.xml, as a sub-element of <cluster> , as in the cluster example in the Tomcat docs.

Report · Dec 23, 2021

Further thoughts:

Could you please share any relevant Tomcat log messages: from isapi_redirect.log, coldfusion-error.log, coldfusion-out.log.
I came across a fix for connector crashes in ColdFusion. The DLL downloads are only applicable to IIS. However, the recommended settings are applicable to IIS as well as Apache:
"For IIS: In the isapi_redirect.properties file, set max_workers_per_lb=number of instances in the cluster.
For Apache: In the mod_jk.conf file, set JkMaxWorkersPerLb=number of instances in the cluster.

The values you set represents the maximum number of instances in the cluster. If you add more clusters, the connectors stop working.

After making the updates, restart the webserver."

Report · Jan 09, 2022

Hi @GuitsBoy , any developments? New insights?

Do you see similarities in the following issue? https://community.adobe.com/t5/coldfusion-discussions/cgi-scope-is-empty-with-coldfusion-2021-on-win...

Report · Jan 10, 2022

Sorry BKBK, I've been meaning to get back into this after the holidays, but I just cant seem to catch a break lately.

I'm not really sure how applicable either of those two suggestions are, since we are not experiencing a crash (connector or otherwise), and I just didnt pick up on any parallels with the second link, since we dont see our sessions get wiped out. We are currently only working with four instances in each local cluster on each box. The clusters form one box to another do not talk to eachother.

I will look into the requested log files as soon as I can get past the current fires, but you know how management can be, its only important if its on fire. Yesterdays problems are tomorrows problems, not todays.

Thanks,
-Tony

Report · Jan 11, 2022

Here is my jvm.config for each of the four instances:

# VM configuration
#
# Where to find JVM, if {java.home}/jre exists then that JVM is used
# if not then it must be the path to the JRE itself

java.home=/opt/ColdFusion2021/jre

#
# If no java.home is specified a VM is located by looking in these places in this
# order:
#
#  1) ../runtime/jre
#  2) registry (windows only)
#  3) JAVA_HOME env var plus jre (ie $JAVA_HOME/jre)
#  4) java.exe in path
#

application.home=/opt/coldfusion10/cfusion1

# Arguments to VM
java.args=-server  -Djava.awt.headless=true -Xms512m -Xmx8000m --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.util.cldr=ALL-UNNAMED --add-opens=java.base/sun.util.locale.provider=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseParallelGC -Djdk.attach.allowAttachSelf=true -Dcoldfusion.home={application.home} -Djava.security.egd=/dev/urandom -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file={application.home}/lib/logging.properties -Dtika.config=tika-config.xml -Djava.locale.providers=COMPAT,SPI -Dsun.font.layoutengine=icu -Dcom.sun.media.jai.disableMediaLib=true -Dcoldfusion.datemask.useDasdayofmonth=true -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dlog4j2.formatMsgNoLookups=true -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib/,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/cfform/jars,{application.home}/bin/cf-osgicli.jar,/opt/class

java.class.path={application.home}/lib/oosdk/lib,{application.home}/lib/oosdk/classes

# Comma separated list of shared library path
java.library.path={application.home}/lib/international

For testing, I just restarted cfusion2 on Node3 while in production. The issue was immediately apparent, and the instances would not restart.

I am not using IIS, these are RHEL8 boxes and Apache/mod_jk, so here is the mod_jk log file:

From the mod_jk.log

[Tue Jan 11 08:47:15 2022] [3354952:139701889312512] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 08:49:04 2022] [3355172:139701864363776] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion3) Writing to client aborted or client network problems
[Tue Jan 11 08:49:04 2022] [3355172:139701864363776] [info] ajp_service::jk_ajp_common.c (3000): (cfusion3) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 08:49:04 2022] [3355172:139701864363776] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion3 is in local error state
[Tue Jan 11 08:49:04 2022] [3355172:139701864363776] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 08:49:04 2022] [3355172:139701864363776] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 08:50:05 2022] [3354952:139701889312512] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion3) Writing to client aborted or client network problems
[Tue Jan 11 08:50:05 2022] [3354952:139701889312512] [info] ajp_service::jk_ajp_common.c (3000): (cfusion3) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 08:50:05 2022] [3354952:139701889312512] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion3 is in local error state
[Tue Jan 11 08:50:05 2022] [3354952:139701889312512] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 08:50:05 2022] [3354952:139701889312512] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 08:50:32 2022] [3354954:139701394601728] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion3) Writing to client aborted or client network problems
[Tue Jan 11 08:50:32 2022] [3354954:139701394601728] [info] ajp_service::jk_ajp_common.c (3000): (cfusion3) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 08:50:32 2022] [3354954:139701394601728] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion3 is in local error state
[Tue Jan 11 08:50:32 2022] [3354954:139701394601728] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 08:50:32 2022] [3354954:139701394601728] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:05:33 2022] [3355172:139700631226112] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion4) Writing to client aborted or client network problems
[Tue Jan 11 09:05:33 2022] [3355172:139700631226112] [info] ajp_service::jk_ajp_common.c (3000): (cfusion4) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:05:33 2022] [3355172:139700631226112] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion4 is in local error state
[Tue Jan 11 09:05:33 2022] [3355172:139700631226112] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:05:33 2022] [3355172:139700631226112] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:16:08 2022] [3355172:139700622833408] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion4) Writing to client aborted or client network problems
[Tue Jan 11 09:16:08 2022] [3355172:139700622833408] [info] ajp_service::jk_ajp_common.c (3000): (cfusion4) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:16:08 2022] [3355172:139700622833408] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion4 is in local error state
[Tue Jan 11 09:16:08 2022] [3355172:139700622833408] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:16:08 2022] [3355172:139700622833408] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:32:42 2022] [3355172:139700648011520] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion1) Writing to client aborted or client network problems
[Tue Jan 11 09:32:42 2022] [3355172:139700648011520] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:32:42 2022] [3355172:139700648011520] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 09:32:42 2022] [3355172:139700648011520] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:32:42 2022] [3355172:139700648011520] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:34:02 2022] [3354953:139701688182528] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion1) Writing to client aborted or client network problems
[Tue Jan 11 09:34:02 2022] [3354953:139701688182528] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:34:02 2022] [3354953:139701688182528] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 09:34:02 2022] [3354953:139701688182528] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:34:02 2022] [3354953:139701688182528] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:34:43 2022] [3355172:139701310707456] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion1) Writing to client aborted or client network problems
[Tue Jan 11 09:34:43 2022] [3355172:139701310707456] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:34:43 2022] [3355172:139701310707456] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 09:34:43 2022] [3355172:139701310707456] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:34:43 2022] [3355172:139701310707456] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] ajp_read_into_msg_buff::jk_ajp_common.c (1654): (cfusion1) receiving data from client failed. Connection aborted or network problems
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] ajp_process_callback::jk_ajp_common.c (2247): (cfusion1) Reading from client aborted or client network problems
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client read error (attempt=1)
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] service::jk_lb_worker.c (1623): unrecoverable error 400, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:36:12 2022] [3354953:139702083991296] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 09:36:51 2022] [3354953:139700589295360] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion1) Writing to client aborted or client network problems
[Tue Jan 11 09:36:51 2022] [3354953:139700589295360] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 09:36:51 2022] [3354953:139700589295360] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 09:36:51 2022] [3354953:139700589295360] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 09:36:51 2022] [3354953:139700589295360] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 10:39:55 2022] [3354954:139701386209024] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1443): (cfusion2) can't receive the response header message from tomcat, tomcat (127.0.0.1:8013) has forced a connection close for socket 42
[Tue Jan 11 10:39:55 2022] [3354954:139701386209024] [error] ajp_get_reply::jk_ajp_common.c (2420): (cfusion2) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Tue Jan 11 10:39:55 2022] [3354954:139701386209024] [info] ajp_service::jk_ajp_common.c (3000): (cfusion2) sending request to tomcat failed (recoverable),  (attempt=1)
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1443): (cfusion2) can't receive the response header message from tomcat, tomcat (127.0.0.1:8013) has forced a connection close for socket 62
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [error] ajp_get_reply::jk_ajp_common.c (2420): (cfusion2) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [info] ajp_service::jk_ajp_common.c (3000): (cfusion2) sending request to tomcat failed (recoverable),  (attempt=1)
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1443): (cfusion2) can't receive the response header message from tomcat, tomcat (127.0.0.1:8013) has forced a connection close for socket 40
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [error] ajp_get_reply::jk_ajp_common.c (2420): (cfusion2) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [info] ajp_service::jk_ajp_common.c (3000): (cfusion2) sending request to tomcat failed (recoverable),  (attempt=2)
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [error] ajp_service::jk_ajp_common.c (3021): (cfusion2) connecting to tomcat failed (rc=0, errors=1, client_errors=12).
[Tue Jan 11 10:39:56 2022] [3355172:139702109169408] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion2 is in local error state
[Tue Jan 11 10:39:58 2022] [3354954:139701386209024] [info] ajp_connection_tcp_get_message::jk_ajp_common.c (1451): (cfusion2) can't receive the response header message from tomcat, network problems or tomcat (127.0.0.1:8013) is down (errno=104)
[Tue Jan 11 10:39:58 2022] [3354954:139701386209024] [error] ajp_get_reply::jk_ajp_common.c (2420): (cfusion2) Tomcat is down or refused connection. No response has been sent to the client (yet)
[Tue Jan 11 10:39:58 2022] [3354954:139701386209024] [info] ajp_service::jk_ajp_common.c (3000): (cfusion2) sending request to tomcat failed (recoverable),  (attempt=2)
[Tue Jan 11 10:39:58 2022] [3354954:139701386209024] [error] ajp_service::jk_ajp_common.c (3021): (cfusion2) connecting to tomcat failed (rc=0, errors=2, client_errors=12).
[Tue Jan 11 10:39:58 2022] [3354954:139701386209024] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion2 is in error state
[Tue Jan 11 10:42:46 2022] [3355172:139702109169408] [info] ajp_process_callback::jk_ajp_common.c (2288): current reuse count is 8 of max reuse connection -1 and total endpoint count 25
[Tue Jan 11 10:44:12 2022] [3354954:139701730146048] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion1) Writing to client aborted or client network problems
[Tue Jan 11 10:44:12 2022] [3354954:139701730146048] [info] ajp_service::jk_ajp_common.c (3000): (cfusion1) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 10:44:12 2022] [3354954:139701730146048] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion1 is in local error state
[Tue Jan 11 10:44:12 2022] [3354954:139701730146048] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 10:44:12 2022] [3354954:139701730146048] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
[Tue Jan 11 10:49:51 2022] [3354952:139701033879296] [info] ajp_process_callback::jk_ajp_common.c (2219): (cfusion4) Writing to client aborted or client network problems
[Tue Jan 11 10:49:51 2022] [3354952:139701033879296] [info] ajp_service::jk_ajp_common.c (3000): (cfusion4) sending request to tomcat failed (unrecoverable), because of client write error (attempt=1)
[Tue Jan 11 10:49:51 2022] [3354952:139701033879296] [info] service::jk_lb_worker.c (1604): service failed, worker cfusion4 is in local error state
[Tue Jan 11 10:49:51 2022] [3354952:139701033879296] [info] service::jk_lb_worker.c (1623): unrecoverable error 200, request failed. Client failed in the middle of request, we can't recover to another instance.
[Tue Jan 11 10:49:51 2022] [3354952:139701033879296] [info] jk_handler::mod_jk.c (3444): Aborting connection for worker=CFCluster
(END)

From the cfusion2 coldfusion-out.log: (restarted instance)

Jan 11, 2022 10:38:44 AM Information [ajp-nio-127.0.0.1-8013-exec-15] - HTTP request completed  {Status Code=200 ,Time taken=25 ms}
Jan 11, 2022 10:38:45 AM Information [ajp-nio-127.0.0.1-8013-exec-27] - Starting HTTP request {URL='http://contentcafe2.btol.com/contentcafe/contentcafe.asmx/Single?userID=XXXXXXXXXX&password=XXXXXXXXXX&key=9781350146440&content=AvailableContent', method='get'}
Jan 11, 2022 10:38:45 AM Information [ajp-nio-127.0.0.1-8013-exec-27] - HTTP request completed  {Status Code=200 ,Time taken=23 ms}
Jan 11, 2022 10:38:45 AM Information [ajp-nio-127.0.0.1-8013-exec-27] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%0D%0A%09%09%09%09%09%09%09%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%09%09%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781350146440%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781350146440%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781350146440%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%09%09%09%09%09%3C%2FContentCafe%3E%0D%0A%09%09%09%09%09%09', method='get'}
Jan 11, 2022 10:38:45 AM Information [ajp-nio-127.0.0.1-8013-exec-27] - HTTP request completed  {Status Code=200 ,Time taken=25 ms}
Jan 11, 2022 10:38:47 AM Information [ajp-nio-127.0.0.1-8013-exec-35] - Starting HTTP request {URL='http://contentcafe2.btol.com/contentcafe/contentcafe.asmx/Single?userID=XXXXXXXXXX&password=XXXXXXXXXX&key=9780531205174&content=AvailableContent', method='get'}
Jan 11, 2022 10:38:47 AM Information [ajp-nio-127.0.0.1-8013-exec-35] - HTTP request completed  {Status Code=200 ,Time taken=21 ms}
Jan 11, 2022 10:38:47 AM Information [ajp-nio-127.0.0.1-8013-exec-35] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%0D%0A%09%09%09%09%09%09%09%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%09%09%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780531205174%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780531205174%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780531205174%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%09%09%09%09%09%3C%2FContentCafe%3E%0D%0A%09%09%09%09%09%09', method='get'}
Jan 11, 2022 10:38:47 AM Information [ajp-nio-127.0.0.1-8013-exec-35] - HTTP request completed  {Status Code=200 ,Time taken=25 ms}
Jan 11, 2022 10:38:49 AM Information [ajp-nio-127.0.0.1-8013-exec-10] - Starting HTTP request {URL='http://contentcafe2.btol.com/contentcafe/contentcafe.asmx/Single?userID=XXXXXXXXXX&password=XXXXXXXXXX&key=9780486494500&content=AvailableContent', method='get'}
Jan 11, 2022 10:38:49 AM Information [ajp-nio-127.0.0.1-8013-exec-10] - HTTP request completed  {Status Code=200 ,Time taken=22 ms}
Jan 11, 2022 10:38:49 AM Information [ajp-nio-127.0.0.1-8013-exec-10] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%0D%0A%09%09%09%09%09%09%09%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%09%09%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780486494500%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780486494500%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780486494500%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%09%09%09%09%09%3C%2FContentCafe%3E%0D%0A%09%09%09%09%09%09', method='get'}
Jan 11, 2022 10:38:49 AM Information [ajp-nio-127.0.0.1-8013-exec-10] - HTTP request completed  {Status Code=200 ,Time taken=25 ms}
Jan 11, 2022 10:38:51 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - Starting HTTP request {URL='http://contentcafe2.btol.com/contentcafe/contentcafe.asmx/Single?userID=XXXXXXXXXX&password=XXXXXXXXXX&key=9781645675020&content=AvailableContent', method='get'}
Jan 11, 2022 10:38:51 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - HTTP request completed  {Status Code=200 ,Time taken=96 ms}
Jan 11, 2022 10:38:51 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%0D%0A%09%09%09%09%09%09%09%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%09%09%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781645675020%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781645675020%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781645675020%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%09%09%09%09%09%3C%2FContentCafe%3E%0D%0A%09%09%09%09%09%09', method='get'}
Jan 11, 2022 10:38:51 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - HTTP request completed  {Status Code=200 ,Time taken=24 ms}
Jan 11, 2022 10:38:52 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - Starting HTTP request {URL='http://contentcafe2.btol.com/contentcafe/contentcafe.asmx/Single?userID=XXXXXXXXXX&password=XXXXXXXXXX&key=9781250156723&content=AvailableContent', method='get'}
Jan 11, 2022 10:38:52 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - HTTP request completed  {Status Code=200 ,Time taken=21 ms}
Jan 11, 2022 10:38:52 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%0D%0A%09%09%09%09%09%09%09%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%09%09%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781250156723%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781250156723%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9781250156723%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%09%09%09%09%09%3C%2FContentCafe%3E%0D%0A%09%09%09%09%09%09', method='get'}
Jan 11, 2022 10:38:52 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - HTTP request completed  {Status Code=200 ,Time taken=27 ms}
Jan 11, 2022 10:39:00 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%0D%0A%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%3C%2FContentCafe%3E', method='get'}
Jan 11, 2022 10:39:00 AM Information [ajp-nio-127.0.0.1-8013-exec-29] - HTTP request completed  {Status Code=200 ,Time taken=32 ms}
Jan 11, 2022 10:39:01 AM Information [ajp-nio-127.0.0.1-8013-exec-36] - Session rotated successfully.
Jan 11, 2022 10:39:08 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - Starting HTTP request {URL='http://contentcafe2.btol.com/ContentCafe/ContentCafe.asmx/XmlString?xmlRequest=%3CContentCafe%20xmlns%3D%22http%3A%2F%2FContentCafe2.btol.com%22%3E%0D%0A%09%09%09%3CRequestItems%20UserID%3D%22XXXXXXXXXX%22%20Password%3D%22XXXXXXXXXX%22%3E%0D%0A%09%09%09%09%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22001%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3EProductDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22002%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3EAnnotationDetail%3C%2FContent%3E%3C%2FRequestItem%3E%0D%0A%09%09%09%09%09%3CRequestItem%20ID%3D%22003%22%3E%0D%0A%09%09%09%09%09%09%3CKey%20Type%3D%22ISBN%22%3E9780134426808%3C%2FKey%3E%0D%0A%09%09%09%09%09%09%3CContent%3ETocDetail%3C%2FContent%3E%0D%0A%09%09%09%09%09%3C%2FRequestItem%3E%0D%0A%09%09%09%09%0D%0A%09%09%09%3C%2FRequestItems%3E%0D%0A%09%09%3C%2FContentCafe%3E', method='get'}
Jan 11, 2022 10:39:08 AM Information [ajp-nio-127.0.0.1-8013-exec-23] - HTTP request completed  {Status Code=200 ,Time taken=25 ms}
Jan 11, 2022 10:39:17 AM Information [ajp-nio-127.0.0.1-8013-exec-25] - Session rotated successfully.
Jan 11, 2022 10:39:57 AM Information [Thread-25] - Monitoring Service stopped.
Jan 11, 2022 10:39:57 AM Information [FelixStartLevel] - Monitoring Service stopped.
Jan 11, 2022 10:39:57 AM Information [Thread-25] - ColdFusion stopped
01/11 10:39:57 [Thread-9] ERROR Could not flush disk cache. Initial cause was /tmp/ehcache_auto_created11567853612919146648diskstore/customcache.index (No such file or directory)
java.io.FileNotFoundException: /tmp/ehcache_auto_created11567853612919146648diskstore/customcache.index (No such file or directory)
        at java.base/java.io.FileOutputStream.open0(Native Method)
        at java.base/java.io.FileOutputStream.open(FileOutputStream.java:298)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:237)
        at java.base/java.io.FileOutputStream.<init>(FileOutputStream.java:187)
        at net.sf.ehcache.store.disk.DiskStorageFactory$IndexWriteTask.call(DiskStorageFactory.java:1103)
        at net.sf.ehcache.store.disk.DiskStorageFactory.unbind(DiskStorageFactory.java:921)
        at net.sf.ehcache.store.disk.DiskStore.dispose(DiskStore.java:664)
        at net.sf.ehcache.store.CacheStore.dispose(CacheStore.java:342)
        at net.sf.ehcache.Cache.dispose(Cache.java:2587)
        at net.sf.ehcache.CacheManager.shutdown(CacheManager.java:1548)
        at coldfusion.cache.ehcache.GenericEhcache$CacheCleanUpAgent.shutdown(GenericEhcache.java:1147)
        at coldfusion.cache.ehcache.GenericEhcache$CacheCleanUpAgent.run(GenericEhcache.java:1112)
        at coldfusion.scheduling.Scheduler.run(Scheduler.java:245)
        at java.base/java.lang.Thread.run(Thread.java:834)
(END)

From the cfusion2 coldfusion-error.log: (restarted instance)


Jan 11, 2022 10:39:52 AM org.apache.catalina.core.StandardServer await
WARNING: Invalid shutdown command [] received
Jan 11, 2022 10:39:52 AM org.apache.catalina.core.StandardServer await
INFO: A valid shutdown command was received via the shutdown port. Stopping the Server instance.
Jan 11, 2022 10:39:52 AM org.apache.coyote.XXXXXXXXXXtractProtocol pause
INFO: Pausing ProtocolHandler ["http-nio-8502"]
Jan 11, 2022 10:39:52 AM org.apache.coyote.XXXXXXXXXXtractProtocol pause
INFO: Pausing ProtocolHandler ["ajp-nio-127.0.0.1-8013"]
Jan 11, 2022 10:39:52 AM org.apache.catalina.core.StandardService stopInternal
INFO: Stopping service [Catalina]
Jan 11, 2022 10:39:52 AM org.apache.catalina.ha.session.JvmRouteBinderValve stopInternal
INFO: JvmRouteBinderValve stopped
Jan 11, 2022 10:39:57 AM org.apache.catalina.ha.session.DeltaManager stopInternal
INFO: Manager [localhost#] expiring sessions upon shutdown
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@3bb9b434]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@739aa022]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@7ee1962d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@50d2582d]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@10cca934]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@381ee69]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@5611f2ce]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@ce75c16]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@24b346df]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@66626154]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@6a1c2957]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@33c22158]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@278a09fd]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@df91c41]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@7864a46b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@73b96ca1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@51535b6f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@7fb8ff07]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@140d4e10]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@4917e706]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@71f346e1]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@3a30f393]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@4ae32c62]) and a value of type [org.apache.xerces.parsers.SAXParser] (value [org.apache.xerces.parsers.SAXParser@3e90c970]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@44b3570b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@415d4b1]) and a value of type [org.apache.xerces.jaxp.SAXParserImpl] (value [org.apache.xerces.jaxp.SAXParserImpl@32eaa74c]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.catalina.loader.WebappClassLoaderBase checkThreadLocalMapForLeaks
SEVERE: The web application [ROOT] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@3c0d391e]) and a value of type [org.apache.xerces.jaxp.SAXParserFactoryImpl] (value [org.apache.xerces.jaxp.SAXParserFactoryImpl@59452bbe]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Jan 11, 2022 10:39:57 AM org.apache.coyote.XXXXXXXXXXtractProtocol stop
INFO: Stopping ProtocolHandler ["http-nio-8502"]
Jan 11, 2022 10:39:57 AM org.apache.coyote.XXXXXXXXXXtractProtocol stop
INFO: Stopping ProtocolHandler ["ajp-nio-127.0.0.1-8013"]
Jan 11, 2022 10:40:02 AM org.apache.catalina.core.AprLifecycleListener lifecycleEvent
INFO: The Apache Tomcat Native library which allows using OpenSSL was not found on the java.library.path: [/opt/ColdFusion2021/cfusion2/bin/../lib:/opt/ColdFusion2021/cfusion2/bin/../lib/_linux64::/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib]
Jan 11, 2022 10:40:03 AM org.apache.coyote.XXXXXXXXXXtractProtocol init
INFO: Initializing ProtocolHandler ["http-nio-8502"]
Jan 11, 2022 10:40:03 AM org.apache.coyote.XXXXXXXXXXtractProtocol init
INFO: Initializing ProtocolHandler ["ajp-nio-127.0.0.1-8013"]
Jan 11, 2022 10:40:03 AM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Catalina]
Jan 11, 2022 10:40:03 AM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet engine: [Apache Tomcat/9.0.50]
Jan 11, 2022 10:40:03 AM org.apache.catalina.ha.tcp.SimpleTcpCluster startInternal
INFO: Cluster is about to start
Jan 11, 2022 10:40:03 AM org.apache.catalina.tribes.transport.ReceiverBase bind
INFO: Receiver Server Socket bound to:[/10.10.240.113:4032]
Jan 11, 2022 10:40:11 AM org.apache.catalina.tribes.util.UUIDGenerator <clinit>
INFO: Creation of SecureRandom instance for UUID generation using [DRBG] took [8,012] milliseconds.
Jan 11, 2022 10:40:11 AM org.apache.catalina.tribes.membership.McastServiceImpl setupSocket
INFO: Setting cluster mcast soTimeout to [500]
Jan 11, 2022 10:40:11 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for [1000] milliseconds to establish cluster membership, start level:[4]
Jan 11, 2022 10:40:11 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4033,{10, 10, 240, 113},4033, alive=1881488532, securePort=-1, UDP Port=-1, id={-4 77 2 66 23 112 72 109 -76 54 0 -107 -68 42 -18 -89 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:11 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4031,{10, 10, 240, 113},4031, alive=1881938108, securePort=-1, UDP Port=-1, id={56 73 -38 27 -6 -117 77 50 -67 -121 57 -92 73 67 -98 -69 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:11 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4034,{10, 10, 240, 113},4034, alive=1881396824, securePort=-1, UDP Port=-1, id={-65 -111 -87 -6 -34 11 65 52 -121 68 19 85 -68 -52 93 52 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:12 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:[4]
Jan 11, 2022 10:40:12 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Sleeping for [1000] milliseconds to establish cluster membership, start level:[8]
Jan 11, 2022 10:40:12 AM org.apache.catalina.tribes.io.BufferPool getBufferPool
INFO: Created a buffer pool with max size:[104857600] bytes of type: [org.apache.catalina.tribes.io.BufferPool15Impl]
Jan 11, 2022 10:40:12 AM java.io.ObjectInputFilter$Config lambda$static$0
INFO: Creating serialization filter from !org.mozilla.**;!com.sun.syndication.**;!org.apache.commons.beanutils.**
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:13 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]
Jan 11, 2022 10:40:13 AM org.apache.catalina.tribes.membership.McastServiceImpl waitForMembers
INFO: Done sleeping, membership established, start level:[8]
Jan 11, 2022 10:40:16 AM org.apache.catalina.util.SessionIdGeneratorBase createSecureRandom
WARNING: Creation of SecureRandom instance for session ID generation using [SHA1PRNG] took [2,603] milliseconds.
Jan 11, 2022 10:40:16 AM org.apache.catalina.ha.session.DeltaManager startInternal
INFO: Register manager [localhost#] to cluster element [Engine] with name [Catalina]
Jan 11, 2022 10:40:16 AM org.apache.catalina.ha.session.DeltaManager startInternal
INFO: Starting clustering manager at [localhost#]
Jan 11, 2022 10:40:16 AM org.apache.catalina.ha.session.DeltaManager getAllClusterSessions
INFO: Manager [localhost#], requesting session state from [org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4031,{10, 10, 240, 113},4031, alive=1881942609, securePort=-1, UDP Port=-1, id={56 73 -38 27 -6 -117 77 50 -67 -121 57 -92 73 67 -98 -69 }, payload={}, command={}, domain={}]]. This operation will timeout if no session state has been received within [60] seconds.
Jan 11, 2022 10:40:21 AM org.apache.catalina.ha.session.DeltaManager waitForSendAllSessions
INFO: Manager [localhost#]; session state sent at [1/11/22 10:40 AM] received in [5,209] ms.

From the cfusion1 coldfusion-error.log: (peer instance)

Jan 11, 2022 10:39:57 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberDisappeared
INFO: Received member disappeared:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=1881769626, securePort=-1, UDP Port=-1, id={121 82 -44 115 1 97 74 -96 -73 -7 -53 -122 117 -79 -34 -29 }, payload={}, command={66 65 66 89 45 65 76 69 88 ...(9)}, domain={}]]
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded
INFO: Replication member added:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=1006, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:21 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=4508, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:21 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=4508, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:21 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:717)
        at org.apache.catalina.ha.session.DeltaManager.sendSessions(DeltaManager.java:1503)
        at org.apache.catalina.ha.session.DeltaManager.handleGET_ALL_SESSIONS(DeltaManager.java:1418)
        at org.apache.catalina.ha.session.DeltaManager.messageReceived(DeltaManager.java:1233)
        at org.apache.catalina.ha.session.DeltaManager.messageDataReceived(DeltaManager.java:997)
        at org.apache.catalina.ha.session.ClusterSessionListener.messageReceived(ClusterSessionListener.java:77)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:821)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.messageReceived(SimpleTcpCluster.java:803)
        at org.apache.catalina.tribes.group.GroupChannel.messageReceived(GroupChannel.java:345)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.messageReceived(TcpFailureDetector.java:118)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.messageReceived(ChannelInterceptorBase.java:96)
        at org.apache.catalina.tribes.group.ChannelCoordinator.messageReceived(ChannelCoordinator.java:288)
        at org.apache.catalina.tribes.transport.ReceiverBase.messageDataReceived(ReceiverBase.java:272)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.drainChannel(NioReplicationTask.java:229)
        at org.apache.catalina.tribes.transport.nio.NioReplicationTask.run(NioReplicationTask.java:103)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:30 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=19514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:30 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=19514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:30 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:30 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=19514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:30 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=19514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:30 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=20514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=20514, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:32 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:32 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:32 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:32 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=21014, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:32 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:33 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=22015, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:33 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=22015, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:33 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=22015, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:33 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=22015, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:33 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:35 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=24515, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:35 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=24515, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:35 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:35 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=24515, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:35 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=24515, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:35 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=26016, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=26016, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:37 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=26516, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member still alive[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=26516, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:37 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:40:39 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberDisappeared
INFO: Received member disappeared:[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=4508, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:39 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector performBasicCheck
INFO: Suspect member, confirmed dead.[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=4508, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=29517, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:40:40 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=29517, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:40:40 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:527)
        at org.apache.catalina.ha.session.DeltaSession.expire(DeltaSession.java:496)
        at org.apache.catalina.ha.session.DeltaSession.isValid(DeltaSession.java:458)
        at org.apache.catalina.session.ManagerBase.processExpires(ManagerBase.java:608)
        at org.apache.catalina.session.ManagerBase.backgroundProcess(ManagerBase.java:591)
        at org.apache.catalina.core.StandardContext.backgroundProcess(StandardContext.java:5684)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1367)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1371)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.processChildren(ContainerBase.java:1371)
        at org.apache.catalina.core.ContainerBase$ContainerBackgroundProcessor.run(ContainerBase.java:1349)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:41:20 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=68533, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:41:20 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=68533, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:41:20 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendChangeSessionId(DeltaManager.java:575)
        at org.apache.catalina.ha.session.DeltaManager.changeSessionId(DeltaManager.java:559)
        at org.apache.catalina.ha.session.DeltaManager.changeSessionId(DeltaManager.java:529)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.changeSessionID(JvmRouteBinderValve.java:328)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.handleJvmRoute(JvmRouteBinderValve.java:293)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.handlePossibleTurnover(JvmRouteBinderValve.java:202)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:178)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=73535, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=73535, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:41:25 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=74036, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=74036, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:41:25 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=74036, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:41:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=74036, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:41:25 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122054, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122054, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122555, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122555, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122555, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=122555, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=146063, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=146063, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:37 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:43 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=152066, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:43 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=152066, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:43 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:49 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=158068, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:49 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=158068, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:49 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164070, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164070, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:55 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164070, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164070, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:55 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164570, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:42:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=164570, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:42:55 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=170572, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=170572, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:01 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:07 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=175574, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:07 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=175574, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:07 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:07 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=176574, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:07 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=176574, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:07 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=182076, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=182076, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=182076, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:13 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=182076, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:13 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:19 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=188079, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:19 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=188079, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:19 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=193581, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:25 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=193581, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:25 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:31 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=200083, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:31 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=200083, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:31 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=205585, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=205585, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:37 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=206085, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:37 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=206085, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:37 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:43 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=212087, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:43 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=212087, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:43 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:49 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=218089, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:49 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=218089, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:49 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:43:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=224091, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:43:55 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=224091, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:43:55 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.session.DeltaManager.send(DeltaManager.java:491)
        at org.apache.catalina.ha.session.DeltaManager.sendCreateSession(DeltaManager.java:481)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:454)
        at org.apache.catalina.ha.session.DeltaManager.createSession(DeltaManager.java:440)
        at org.apache.catalina.connector.Request.doGetSession(Request.java:3100)
        at org.apache.catalina.connector.Request.getSession(Request.java:2500)
        at org.apache.catalina.connector.RequestFacade.getSession(RequestFacade.java:908)
        at coldfusion.runtime.AppHelper.setupJ2eeSessionScope(AppHelper.java:1105)
        at coldfusion.runtime.AppHelper.setupSessionScope(AppHelper.java:1205)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:504)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
        at coldfusion.CfmServlet.service(CfmServlet.java:231)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:228)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:190)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:163)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78)
        at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:329)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=229593, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=229593, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:44:01 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=229593, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=229593, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:44:01 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Received memberDisappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=230094, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]] message. Will verify.
Jan 11, 2022 10:44:01 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared
INFO: Verification complete. Member already disappeared[org.apache.catalina.tribes.membership.MemberImpl[tcp://{10, 10, 240, 113}:4032,{10, 10, 240, 113},4032, alive=230094, securePort=-1, UDP Port=-1, id={-72 125 -6 -88 123 -19 66 27 -122 -29 119 30 -118 122 9 100 }, payload={}, command={}, domain={}]]
Jan 11, 2022 10:44:01 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032;
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.XXXXXXXXXXtractProcessorLight.process(XXXXXXXXXXtractProcessorLight.java:65)
        at org.apache.coyote.XXXXXXXXXXtractProtocol$ConnectionHandler.process(XXXXXXXXXXtractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

Further Testing:

Adding JkMaxWorkersPerLB 4 to the mod_jk.conf file did not seem to help.

Moving the manager element from context.xml (stock) to server.xml (as per tomcat docs) did not help. It also seems to go against most information that states the condition was improved when they moved it from server.xml to context.xml.

Adding <distributable /> to the end of the <web-app> element in web.xml did not help, but also does not hurt. Most people say to use it, so I left it.

Completely changing from DeltaManager to BackupManager does indeed seem to help considerably. It will take more testing to know if it is a 100% fix, and if there are any negative repercussions from using it.

Report · Jan 12, 2022

Thanks for sharing that, @GuitsBoy .

A suggestion: to make your last post more readable, include the contents of jvm.config and logs as attachments instead. To achieve this, proceed as follows:

Click on the button to edit your last post ("More" > "Edit Reply").
Copy the jvm.config and log content from your post into text files.
Delete the jvm.config and log content from the body of your post.
Include the text files as attachment.
Re-post the edited post.

Report · Jan 12, 2022

application.home=/opt/coldfusion10/cfusion1

I don't understand. What's ColdFusion10 doing here?

Report · Jan 12, 2022

-Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2

I would delete that, assuming you're on Java 11.0.13. That is because TLS1.0 and TLS1.1 are disabled by default - and TLS1,2 is enabled by default - from JDK11.0.11 and JDK1.8.0_291 onwards. ( https://www.petefreitag.com/item/916.cfm )

TLS1.0 and 1.1 are unsafe. If you really must use them, then you will have to:

enable them in java.security.

use both TLS flags in jvm.config:

-Dhttps.protocols=TTLSv1,TLSv1.1,TLSv1.2,TLSv1.3 
-Djdk.tls.client.protocols=TLSv1,TLSv1.1,TLSv1.2,TLSv1.3

Still on the subject of jvm.config, I missed the following two in your config:

-Djava.net.preferIPv4Stack=true
-Dfile.encoding=UTF-8

Report · Jan 12, 2022

Hi BKBK, I dont seem to have either an "Edit" option nor a "More" option, so I will attach the files here instead.

The "application.home=/opt/coldfusion10/cfusion1" entry is indeed erronious, and was copied over when we imported the settings from CF10. Our development machine has had these updated, however our QC machine shows the non-existant path, and it does not seem to negatively affect anything. From memory, this may have been added when we were attempting to iron out a license/key issue for some add-on software we purchased, but it's been a while. Regardless, we no longer use the software, and this directory does not exist on the new hardware.

The TLS 1.0 and 1.1 flags were added because we need to talk to a vendor's legacy system which required older calls to their API which did not support TLS 1.2. They are aware of he issue, and have been asked to update their system, but I have no idea the status of their project. I dont believe we are transferring any sensitive data over that connection. Since this is for cfhttp and API calls only, perhaps we only need the tls.client.protocols instead of the setting we currently have? Our TLS is offloaded to hardware load balancers and is not handled by java, CF or Apache.

I'm not sure about the preferIPv4Stack=true bit, but perhaps that is related to us having to comment out ::1 in the /etc/hosts file. I'm also not sure what the file.encoding would affect. Are these simply best practices, that have been implemented over the years? I can certainly requiest that we throw these additions in during a future release / regression test.

Anyway, I believe these jvm.config settings should only affect CF behavior once up and running, but wouldnt affect tomcat replication on the back end, would it? Again, the issue seems to be load dependant, since our dev/qc servers are not affected, and production machines having been removed from traffic restart cleanly without trouble. Also, moving from deltaManager to backupManager also seems to fix the problem. I think this really makes session replication a prime suspect.

One additional thing that I go back and forth on, is that the CF instances share the same names across three different physical boxes (cfusion1 - cfusion4), even though these clusters are using different multicast addresses and ports. I have not yet tried to simply change each cluster's name to something unique across all 12 instances. I doubt this is the issue, but figured it was worth mentioning.

Thanks again for your thoughts on this.
-Tony

Report · Jan 13, 2022

The "application.home=/opt/coldfusion10/cfusion1" entry is indeed erronious, and was copied over when we imported the settings from CF10. Our development machine has had these updated, however our QC machine shows the non-existant path, and it does not seem to negatively affect anything. From memory, this may have been added when we were attempting to iron out a license/key issue for some add-on software we purchased, but it's been a while. Regardless, we no longer use the software, and this directory does not exist on the new hardware.

By @GuitsBoy

This is NOT a negligible issue. The setting application.home=/opt/coldfusion10/cfusion1 means that /opt/coldfusion10/cfusion1 is the location of the application intended to use the current Java engine. The library paths and classpaths are all relative to this location.

The TLS 1.0 and 1.1 flags were added because we need to talk to a vendor's legacy system which required older calls to their API which did not support TLS 1.2. They are aware of he issue, and have been asked to update their system, but I have no idea the status of their project. I dont believe we are transferring any sensitive data over that connection. Since this is for cfhttp and API calls only, perhaps we only need the tls.client.protocols instead of the setting we currently have? Our TLS is offloaded to hardware load balancers and is not handled by java, CF or Apache.

Fair enough. However, as I said, in recent Java versions, TLS 1.0 and 1.1 are blocked. If you want to use them, you will have to unblock the block in java.security. If, as you say, you don't need TLS at ColdFusion's level, then remove the TLS flags from the jvm.config. There will then be one less spanner in the works.

I'm not sure about the preferIPv4Stack=true bit, but perhaps that is related to us having to comment out ::1 in the /etc/hosts file. I'm also not sure what the file.encoding would affect. Are these simply best practices, that have been implemented over the years?

Yes.

Anyway, I believe these jvm.config settings should only affect CF behavior once up and running, but wouldnt affect tomcat replication on the back end, would it?

Goor question. I honestly can't say. However, what if Tomcat depends on libraries that are expected to come into ColdFusion via the libPath and classpath defined in jvm.config? If so, then we're screwed, because the libpath and classpath are based on

application.home=/opt/coldfusion10/cfusion1

Again, the issue seems to be load dependant, since our dev/qc servers are not affected, and production machines having been removed from traffic restart cleanly without trouble. Also, moving from deltaManager to backupManager also seems to fix the problem. I think this really makes session replication a prime suspect.

Interesting point. I am looking into it.

Report · Jan 13, 2022

Just for info: I am still studying the Tomcat logs.

Report · Jan 13, 2022

Hi BKBK -

This is NOT a negligible issue. The setting application.home=/opt/coldfusion10/cfusion1 means that /opt/coldfusion10/cfusion1 is the location of the application intended to use the current Java engine. The library paths and classpaths are all relative to this location.

While I don't disagree that thapplication.home variable is an important one, it's undoubtedly being declared elsewhere, superceding the erronious setting in the jvm.config. This is very fortunate, since otherwise nothing would be working at all. In fresh installs of CF2021 without importing settings from a previous version, the application.home variable is not set in the jvm.config, so it must be declared elsewhere.

While I do plan to remove the incorrect setting from the jvm.config ASAP, here is the output from a px -ax showing the correct class paths.

3734326 ?        Sl    30:15 /opt/ColdFusion2021/jre/bin/java -classpath /opt/ColdFusion2021/cfusion1/bin/../runtime/bin/tomcat-juli.jar:/opt/ColdFusion2021/cfusion1/bin/../bin/cf-bootstrap.jar:/opt/ColdFusion2021/cfusion1/bin/../bin/cf-startup.jar:/opt/ColdFusion2021/cfusion1/bin/../runtime/lib/*:/opt/ColdFusion2021/cfusion1/bin/../lib/oosdk/lib:/opt/ColdFusion2021/cfusion1/bin/../lib/oosdk/lib/*:/opt/ColdFusion2021/cfusion1/bin/../lib/oosdk/classes:/opt/ColdFusion2021/cfusion1/bin/../lib/oosdk/classes/*: -server -Djava.awt.headless=true -Xms512m -Xmx8000m --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.util.cldr=ALL-UNNAMED --add-opens=java.base/sun.util.locale.provider=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseParallelGC -Djdk.attach.allowAttachSelf=true -Dcoldfusion.home=/opt/ColdFusion2021/cfusion1/bin/.. -Djava.security.egd=/dev/urandom -Duser.language=en -Dcoldfusion.rootDir=/opt/ColdFusion2021/cfusion1/bin/.. -Dcoldfusion.libPath=/opt/ColdFusion2021/cfusion1/bin/../lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file=/opt/ColdFusion2021/cfusion1/bin/../lib/logging.properties -Dtika.config=tika-config.xml -Djava.locale.providers=COMPAT,SPI -Dsun.font.layoutengine=icu -Dcom.sun.media.jai.disableMediaLib=true -Dcoldfusion.datemask.useDasdayofmonth=true -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dlog4j2.formatMsgNoLookups=true -Dcoldfusion.classPath=/opt/ColdFusion2021/cfusion1/bin/../lib/updates,/opt/ColdFusion2021/cfusion1/bin/../lib/,/opt/ColdFusion2021/cfusion1/bin/../gateway/lib/,/opt/ColdFusion2021/cfusion1/bin/../wwwroot/WEB-INF/cfform/jars,/opt/ColdFusion2021/cfusion1/bin/../bin/cf-osgicli.jar,/opt/class com.adobe.coldfusion.bootstrap.Bootstrap -start

And since /opt/coldfusion10 does not even exist on the current hardware, I would expect a multitude of errors to have presented themselves by now. I'll go ahead and comment out the offending line, but I dont expect our condition to improve with regards to the restart issue. Ill post back if things do behave differently.

Thanks,
-Tony

Report · Jan 13, 2022

Hi BKBK, I dont seem to have either an "Edit" option nor a "More" option, so I will attach the files here instead.

By @GuitsBoy

Hmm. I meant the following button, underneath your post:

Report · Jan 13, 2022

Sorry, but I simply dont have that option

A little reading tells me that Adobe had a spam problem, and simply turned off the edit option for users who have not met some threshold.

Seems a little risky to not at least provide a delete function, in case I were to accidentally post sensitive info or passwords, etc.

Report · Jan 13, 2022

I removed the incorrect line from the jvm.config and restarted all instances.

Node3, which is running BackupManager restarted cleanly on each of the four insatnces, without trouble.

Node1 and Node2 are still running he stock DeltaManager, and they were still extremely tempermental, in a couple cases, taking as many as 8 attempts to restart un order to actually come up correctly. I even shut down all instances and lit them back up one at a time, disrupting traffic tothat node momentarily. The first two came back easily, but the next to hung and required serveral restarts until they correctly came up.

I'm strongly considering just switching the other two nodes to BackupManager and calling it a day, but its still frustrating that DeltaManager is not working.

Since DeltaManager is all-to-all replication, I could see it being a timeout somewhere. Unfortunately I''ve increased every timeout I can think of, without it making a difference.

Report · Jan 14, 2022

I removed the incorrect line from the jvm.config and restarted all instances.

Node3, which is running BackupManager restarted cleanly on each of the four insatnces, without trouble.

Node1 and Node2 are still running he stock DeltaManager, and they were still extremely tempermental, in a couple cases, taking as many as 8 attempts to restart un order to actually come up correctly. I even shut down all instances and lit them back up one at a time, disrupting traffic tothat node momentarily. The first two came back easily, but the next to hung and required serveral restarts until they correctly came up.

I'm strongly considering just switching the other two nodes to BackupManager and calling it a day, but its still frustrating that DeltaManager is not working.

Since DeltaManager is all-to-all replication, I could see it being a timeout somewhere. Unfortunately I''ve increased every timeout I can think of, without it making a difference.

By @GuitsBoy

Admirable patience and debugging skills there, @GuitsBoy . Thanks for your answers to the questions I asked.

Your suspicion of a timeout somewhere concides with my own thoughts: bad node. As DeltaManager is all-to-all, Tomcat will attempt to replicate sessions with every node that it detects. It takes just one bad one.

Apparently, in your environment, there is at least one bad node. By which I mean a Tomcat "member" that is very often either inactive, inaccessible or not functioning properly. For example, the member on the socket 10.10.240.113:4032

SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032

That was in the cfusion1 coldfusion-error.log. Throughout, that member is either unavailable or keeps disappearing.

I suspect a connection problem. Could there perhaps be conflict at a port?

Verify whether the <Membership> or <Receiver> ports of any two CF instances are the same. Also, if you want to do something about timeouts you can tweak some of the other attributes,

<Membership className="org.apache.catalina.tribes.membership.McastService"
          				port="45501"
          				address="228.0.0.1"
          				dropTime="3000"
          				frequency="500"/>
			<Receiver className="org.apache.catalina.tribes.transport.nio.NioReceiver"	            
        				address="auto"
        				autoBind="100"
        				port="4011"
        				selectorTimeout="5000"
        				maxThreads="6"/>

Also look into the settings of the connector conf and server.xml of each CF instance. Do the connector port, timeout, maxThreads, etc. values match exactly those in server.xml?

Something else: you said you did test by placing the <manager> in server.xml instead of context.xml? (as in the docs: https://tomcat.apache.org/tomcat-9.0-doc/cluster-howto.html ) That suggestion goes hand in hand with placing <distributable /> in web./xml, which I presume you also did. But then, there were these warnings on cfusion2 coldfusion-error.log:

Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.session.ClusterSessionListener messageReceived
WARNING: Context manager doesn't exist:[localhost#]

My interpretation of the Tomcat docs is that you need to do one of the following (but NOT both):

Place <distributable/> in web.xml and use the default <Manager> specified in server.xml or,
Add a <Manager> inside context.xml

If after testing the new suggestions you continued to have server issues, then it just might be a server bug. Given the state of your errors, I would then switch from DeltaManager to BackupManager.

Tomcat guru Christopher Schultz says of the two managers:

"BackupManager scales much better than the DeltaManager, anyway. For my money, DeltaManager is a good way to get things proven-out and configured in a dev/test environment, while BackupManager is much more appropriate for production."

Report · Jan 14, 2022

Thanks BKBK.

I dont believe we have a "bad node" per tomcat, since we experience the same issue across three separate physical servers. A misconfiguration is certainly possible, but odds are very low that a tomcat instance goes bad on each one consistently.

The Faulty Member message is logged either when I manually shut down the coldfusion instance myself, or if the coldfusion instance fails to come up correctly (hang on startup). We dont see those errors during normal operation with four healthy instances per cluster.

As for tomcat port configuration, here is our topology:

Node1 (10.10.240.111)

Membership Address: 228.0.0.1

Membership Port: 45501

Membership Drop Time: 3000

Membership Frequency: 500

cfusion1 receiver port: 4011

cfusion2 receiver port: 4012

cfusion3 receiver port: 4013

cfusion4 receiver port: 4014

Node2 (10.10.240.112)

Membership Address: 228.0.0.2

Membership Port: 45502

Membership Drop Time: 3000

Membership Frequency: 500

cfusion1 receiver port: 4021

cfusion2 receiver port: 4022

cfusion3 receiver port: 4023

cfusion4 receiver port: 4024

Node3 (10.10.240.113)

Membership Address: 228.0.0.3

Membership Port: 45503

Membership Drop Time: 3000

Membership Frequency: 500

cfusion1 receiver port: 4031

cfusion2 receiver port: 4032

cfusion3 receiver port: 4033

cfusion4 receiver port: 4034

As you can see, Ive changed both the multicast address and port to be unique for each of the three clusters. Then Ive changd each of the receiver ports to be unique amont the 12 instances.

As for the distributable element and the manager element, Ive tried both with and without distributable for both locations of the manager element without any difference. Currently, we have the distributable element in the wwwroot/WEB-INF/web.xml only (stock form) and the manager resides in server.xml

Yes, it seems that BackupManager is better for high traffic sites. Ive read that DeltaManager is good to about four instances, and beyond that BackupManager is the better option. We are right at four instances, however our load is quite low. The apache load over the last three days is 2.9 requests per second on each server, and I believe that includes all requests (images, html files, static content) and not just CFML pages. As you can see, the load for each instance is quite low. I dont know why delta manager would have trouble here.

All the same, it looks like BackupManager is indeed the best option here. The one node has been running for a few days with BackupManager and so far no issues.

Thanks,
-Tony

Report · Jan 16, 2022

Hi @GuitsBoy ,

Thanks for explaining and for sharing the settings.

I dont believe we have a "bad node" per tomcat, since we experience the same issue across three separate physical servers. A misconfiguration is certainly possible, but odds are very low that a tomcat instance goes bad on each one consistently.

By @GuitsBoy

I was referring to a network node (or Tomcat member), not to a server or ColdFusion instance. I saw in the logs that a member may disappear shortly after coming on. For example,

Jan 11, 2022 10:39:57 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberDisappearedI... tcp://{10, 10, 240, 113}:4032...
Jan 11, 2022 10:40:12 AM org.apache.catalina.ha.tcp.SimpleTcpCluster memberAdded... tcp://{10, 10, 240, 113}:4032,...
Jan 11, 2022 10:40:21 AM org.apache.catalina.tribes.group.interceptors.TcpFailureDetector memberDisappeared... tcp://{10, 10, 240, 113}:4032...

I am having a look at the rest of the settings you've shared.

Report · Jan 18, 2022

I was referring to a network node (or Tomcat member), not to a server or ColdFusion instance. I saw in the logs that a member may disappear shortly after coming on. For example,

I blieve a CF instance is for all intents and purposes, a tomcat node. We are not running any additional tomcat installations on these machines. The four coldfusion instances coincide with the four tomcat nodes, and match the cluster multicats address, and receiver port.

The error message youve identified shows up when we manually restart a cfusion instance, in thise case, it looks to be cfusion2 on node3 (instance 2 on physical-server 3). The first two messages are the server going away on stop, then coming back on start. The last message does in fact seem to coincide with a CF instance that hangs on startup. Sucessful startups in the peers' log files only show the memberDissapeared and memberAdded lines. That led us to look for timeouts of 3000ms, but increasing those that we found did not seem to improve the situation.

The rest of that last error look like this:

Jan 11, 2022 10:40:28 AM org.apache.catalina.ha.tcp.SimpleTcpCluster send
SEVERE: Unable to send message through cluster sender.
org.apache.catalina.tribes.ChannelException: Operation has timed out([3000] ms.).; Faulty members:tcp://{10, 10, 240, 113}:4032; 
        at org.apache.catalina.tribes.transport.nio.ParallelNioSender.sendMessage(ParallelNioSender.java:113)
        at org.apache.catalina.tribes.transport.nio.PooledParallelSender.sendMessage(PooledParallelSender.java:51)
        at org.apache.catalina.tribes.transport.ReplicationTransmitter.sendMessage(ReplicationTransmitter.java:65)
        at org.apache.catalina.tribes.group.ChannelCoordinator.sendMessage(ChannelCoordinator.java:83)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.MessageDispatchInterceptor.sendMessage(MessageDispatchInterceptor.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.interceptors.TcpFailureDetector.sendMessage(TcpFailureDetector.java:93)
        at org.apache.catalina.tribes.group.ChannelInterceptorBase.sendMessage(ChannelInterceptorBase.java:89)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:280)
        at org.apache.catalina.tribes.group.GroupChannel.send(GroupChannel.java:231)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:724)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:708)
        at org.apache.catalina.ha.tcp.SimpleTcpCluster.send(SimpleTcpCluster.java:695)
        at org.apache.catalina.ha.tcp.ReplicationValve.send(ReplicationValve.java:531)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendMessage(ReplicationValve.java:517)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendSessionReplicationMessage(ReplicationValve.java:499)
        at org.apache.catalina.ha.tcp.ReplicationValve.sendReplicationMessage(ReplicationValve.java:411)
        at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:340)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:373)
        at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:462)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:893)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1723)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.base/java.lang.Thread.run(Thread.java:834)

So yes, if you mean that the CF server instance is immediately failing on startup, that is our interpretation as well.

Report · Jan 20, 2022

I was referring to a network node (or Tomcat member), not to a server or ColdFusion instance. I saw in the logs that a member may disappear shortly after coming on. For example,

I blieve a CF instance is for all intents and purposes, a tomcat node.

By @GuitsBoy

I think we're speaking at cross purposes.

From what I've seen so far, your CF instances are fine. But it is possible for a CF instance that is alive and well, and capable of serving requests, to drop out of a cluster. That is, to no longer be a cluster member.

By node or member I meant multicast-socket. After all, in the context of the problem we're trying to solve, a node simply consists of a multicast address and port.

Now, on to a new idea.

New hypothesis: what if the following sequence of events is happening in your set-up

A member D of the cluster is down or fails to respond, for whatever reason;
A member S sends information to the rest of the members. However, D is unable to respond.
As a result of the option channelSendOptions="6", S is blocked until all of the receiving nodes, including D, have acknowledged that they have received and processed the message.

If so, then a possible solution is to:

Switch from channelSendOptions="6" (equivalent: channelSendOptions="sync,use_ack" )
to
channelSendOptions="2" (equivalent: channelSendOptions="use_ack" )

Reference: https://tomcat.apache.org/tomcat-9.0-doc/config/cluster.html#SimpleTcpCluster_Attributes

Adobe Community

Clustered CF Instances Hang, Difficult to Restart