coldfusion.runtime.CustomException: Could not initialize class coldfusion.runtime.SessionTracker

Report · Dec 20, 2017

Do you have any ideas about fixing the coldfusion.runtime.SessionTracker exception? Have you encountered it before? We have, unfortunately.

Our system consists of ColdFusion 11 Update 13 and IIS.

"Error","ajp-bio-8012-exec-4","12/15/17","06:45:30","educationLive","Could not initialize class coldfusion.runtime.SessionTracker The specific sequence of files included or processed is: E:\wwwroot\www_education_nl\htdocs\education\Facade.cfc, line: 493 "

at coldfusion.tagext.lang.ThrowTag.doStartTag(ThrowTag.java:142)

at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:3000)

at cfApplication2ecfc1126640006$funcONERROR.runFunction(E:\wwwroot\www_education_nl\htdocs\Application.cfc:493)

at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:487)

at coldfusion.filter.SilentFilter.invoke(SilentFilter.java:47)

at coldfusion.runtime.UDFMethod$ReturnTypeFilter.invoke(UDFMethod.java:420)

at coldfusion.runtime.UDFMethod$ArgumentCollectionFilter.invoke(UDFMethod.java:383)

at coldfusion.filter.FunctionAccessFilter.invoke(FunctionAccessFilter.java:95)

at coldfusion.runtime.UDFMethod.runFilterChain(UDFMethod.java:334)

at coldfusion.runtime.UDFMethod.invoke(UDFMethod.java:231)

at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:643)

at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:432)

at coldfusion.runtime.TemplateProxy.invoke(TemplateProxy.java:402)

at coldfusion.runtime.AppEventInvoker.invoke(AppEventInvoker.java:108)

at coldfusion.runtime.AppEventInvoker.onError(AppEventInvoker.java:491)

at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:552)

at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:42)

at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)

at coldfusion.filter.PathFilter.invoke(PathFilter.java:153)

at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:94)

at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)

at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)

at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

at coldfusion.xml.rpc.CFCServlet.invoke(CFCServlet.java:156)

at coldfusion.xml.rpc.CFCServlet.doPost(CFCServlet.java:348)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)

at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)

at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)

at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:134)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doNext(FusionReactorRequestHandler.java:763)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doHttpServletRequest(FusionReactorRequestHandler.java:351)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.doFusionRequest(FusionReactorRequestHandler.java:214)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorRequestHandler.handle(FusionReactorRequestHandler.java:800)

at com.intergral.fusionreactor.j2ee.filter.FusionReactorCoreFilter.doFilter(FusionReactorCoreFilter.java:36)

at sun.reflect.GeneratedMethodAccessor93.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intergral.fusionreactor.j2ee.filterchain.WrappedFilterChain.doFilter(WrappedFilterChain.java:71)

at sun.reflect.GeneratedMethodAccessor92.invoke(Unknown Source)

at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:498)

at com.intergral.fusionreactor.agent.filter.FusionReactorStaticFilter.doFilter(FusionReactorStaticFilter.java:54)

at com.intergral.fusionreactor.agent.pointcuts.NewFilterChainPointCut$1.invoke(NewFilterChainPointCut.java:41)

at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java)

at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)

at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)

at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)

at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)

at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)

at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:218)

at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:333)

at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:466)

at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:197)

at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)

at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:316)

at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

at java.lang.Thread.run(Thread.java:748)

Report · Dec 20, 2017

Just out of curiosity, what is on line 493 of Facade.cfc? Are you trying to set a session variable?

I've not seen this error message, before. smh.. I'll Google around a bit..

V/r,

^ _ ^

Report · Dec 20, 2017

Thanks, @WolfShade.

Line 493 is not in the Facade.cfc, but in Application.cfc. We get the same line number with every CFC for which the error occurs. And there are plenty.

Line 493 is simply a throw in onError in Application.cfc:

public void function onError(required Any Exception, String EventName="") output="false"

{

/* some unrelated code */

/* Line 493 */ throw(message="#Exception.message#", detail="#Exception.detail#", extendedinfo="#Exception.extendedInfo#", errorcode="#Exception.errorCode#");

}

This implies that when ColdFusion processed the CFCs, it "Could not initialize class coldfusion.runtime.SessionTracker".

Report · Dec 20, 2017

Are you using CFTRY/CFCATCH around any of the functions in the CFC?

V/r,

^ _ ^

Report · Dec 21, 2017

WolfShade wrote
Are you using CFTRY/CFCATCH around any of the functions in the CFC?

In some, yes.

It is clear that the error occurs when ColdFusion is processing the CFCs in the onCFCRequest eventhandler in Application.cfc. It then seems to be unable to initialize the SessionTracker.

Report · Dec 20, 2017

Just curious. If you create a new page with the following, what happens?

<cfset sessionTracker = createObject("java","coldfusion.runtime.SessionTracker")>
<cfdump var="#sessionTracker#">

V/r,

^ _ ^

Report · Dec 21, 2017

WolfShade schreef

Just curious. If you create a new page with the following, what happens?

<cfset sessionTracker = createObject("java","coldfusion.runtime.SessionTracker")> <cfdump var="#sessionTracker#">

I should add that the error only occurs in production. So it's futile for me to create above in development where there is no such error. there is more information relating to the issue. I'll post it next.

Report · Dec 21, 2017

We use a load-balancer.

There are 4 to 6 nodes (ColdFusion instances) per cluster.

The relevant settings for each instance are:

server.xml

</Cluster>

</Service>

</Server>

context.xml

</Context>

Report · Dec 21, 2017

Sadly, I am not much of an admin for CF, and know nothing of clustering, so I don't think I'd be of any help in this one.

V/r,

^ _ ^

Report · Dec 21, 2017

So, this is just a guess, but I think it's the combination of the CF cluster management and the fact that you're also using Fusion Reactor. I'd temporarily disable Fusion Reactor and see if that makes a difference.

Dave Watts, CTO, Fig Leaf Software

Dave Watts, Eidolon LLC

Report · Dec 22, 2017

Thanks, @Dave Watts.

Please let me know why you think the combination of ColdFusion cluster management and FusionReactor might be the problem. I would imagine that the combination is used widely in ColdFusion production environments. Yet I have been unable to find a single example of the error on the web.

Report · Dec 22, 2017

I don't think it is actually that common a combination. In my own limited personal experience, I don't see very many people using CF clustering at all, really. It's entirely possible that this is just a bug in the latest CF and doesn't even have anything to do with Fusion Reactor. I use load balancing with most of our customer solutions, but have steered clear of using CF clustering with it, as it requires each CF instance to talk to the other CF instances in the cluster, and that seems like it would work well enough with two or three, but would have a higher likelihood of failure as you add more instances. There are lower-tech ways to share "session" data, and those have worked well enough for me. So, I'm kind of curious about how many people actually use CF clustering in the first place. I do see lots of people using Fusion Reactor, though.

The onRequest method in Application.cfc is kind of odd, in that it effectively replaces the request itself with the output of the function call, unlike onRequestStart and onRequestEnd which simply affect the current request. The onCFCRequest method seems similar in that way, and maybe that is interfering with the normal session tracking functionality regardless of whatever code you put in the method.

Dave Watts, CTO, Fig Leaf Software

Dave Watts, Eidolon LLC

Report · Dec 26, 2017

ColdFusion Enterprise of course has a history of integrating clustering. This goes back to tools like ClusterCATS. I am therefore presuming that clustering remains a practical solution for load-balancing and availability.

https://forums.adobe.com/people/Dave+Watts wrote
The onRequest method in Application.cfc is kind of odd, in that it effectively replaces the request itself with the output of the function call, unlike onRequestStart and onRequestEnd which simply affect the current request. The onCFCRequest method seems similar in that way, and maybe that is interfering with the normal session tracking functionality regardless of whatever code you put in the method.

You echo a thought I had. I am looking into the session idea some more.

Report · Feb 19, 2018

coldfusion.runtime.CustomException: Could not initialize class coldfusion.runtime.SessionTracker

I have found the cause of this rare exception. The exception indicates that the instance has had a bad start. It is analogous to a sprinter experiencing a false start at the blocks. The consequences persist throught the race.

Thus, ColdFusion continues to log the exception (in coldfusion-out.log) every other minute, until the instance is stopped or restarted.

The exception usually occurs just after onServerStart.

We also discovered that it is related to the following ColdFusion error (see, for example, coldfusion-error.log):

"The Monitoring service is not available. This exception is usually caused by service startup failure. Check your server configuration."

...

Jan 30, 2018 8:17:28 AM org.apache.catalina.core.StandardWrapperValve invoke

SEVERE: Servlet.service() for servlet [CfmServlet] in context with path [] threw exception [ROOT CAUSE:

coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Monitoring service is not available.

at coldfusion.server.ServiceFactory.getMonitoringService(ServiceFactory.java:223)

at coldfusion.CfmServlet.service(CfmServlet.java:199)

at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

...

] with root cause

javax.servlet.ServletException: ROOT CAUSE:

coldfusion.server.ServiceFactory$ServiceNotAvailableException: The Monitoring service is not available.

at coldfusion.server.ServiceFactory.getMonitoringService(ServiceFactory.java:223)

at coldfusion.CfmServlet.service(CfmServlet.java:199)

This, too, is logged continually.

Together, these errors eventually lead to a rise in CPU use by the ColdFusion instance. The end result is unstable behaviour by the instance and 100% CPU use, requiring a restart.

In short, the cause is a false start of the instance, the solution a restart.

Report · Feb 20, 2018

BKBK, I appreciate that you're sharing your observation and conclusion on this: that in your case the bottom line is that a restart of the instance "fixed" the problem. But the only explanation offered really is that the instance had some sort of "false start". The troubleshooter in me wouldn't want to leave it at that. I'd want to know what THAT really meant. 🙂 And I suspect you and others here may also still wonder.

So as I revisit all that was shared in this thread (as indeed I had read it as responses were offered over the weeks), I have some more thoughts, if you want to dig in more.

First, do you have any evidence of whether these errors were happening on EVERY request? Because it could be that they were happening only on SOME requests. (We can't know just from the log snippets you shared here, but you may know.) And also, was it in all apps or only some?

I realize you may not be able to say now. But if this happens to you (or anyone else finding this) again in the future, before just giving up and restarting CF, it could be helpful if you could check if ANY requests at all can run. Of course a challenge may be if everyone you TRY fails, but that doesn't really mean that all that COULD be running would fail. There may even be ones running that you are not trying.
So it would help to see if CF can report if others ARE running, without error. For instance, if you had the "cf metrics" enabled, can you see if a larger number of requests is completing than are generating errors? Or if you have the CF Server Monitor or FusionReactor, or SeeFusion enabled, can you see requests running? Are some failing or are ALL of those failing? FWIW, FR logs information about each request and all requests in aggregate, without any need of configuration, which could be useful after a restart. And your web server logs might also track what requests got errors, vs those that did not.
And is the error happening for all apps on the server, once it starts? Or might it be limited to some apps?

Indeed, here's another test that could be useful: what if you created a test page in its own folder with a blank application.cfm. Would that page "work"? What if the page added enabling of sessionmanagement? (And even if you were in a panic to get CF back up and couldn't do this testing, you could have a scheduled task or probe or monitoring tool calling it regularly, and review the results in the CF or FR logs.)
Finally, could it be perhaps that only SOME kinds of requests even to the same page or app might be failing with this error, while others are not? If so, what characteristics do those requests have? For instance, might they be ones where there are no session cookies coming in, so that CF is creating new sessions? or is there anything about the incoming session cookies that may be amiss, perhaps if they may be tripping over the session fixation protection? Or causing a session rotation? and so on.

These sorts of info could provide useful diagnostics to help get close to perhaps really solving what is amiss.

One last thing: in one of the messages above, you said that "It is clear that the error occurs when ColdFusion is processing the CFCs in the onCFCRequest eventhandler in Application.cfc". I'd be curious what led to that conclusion. I don't see any other reference to oncfcrequest before that note on Dec 21. It's not in the stack trace, which instead refers to an onerror method--and I would argue that that's simply indicating that you had an onerror method intercepting and throwing the this error.

BTW, if you had the error handler dump the complete error scope, you would see the tagcontext, which would show what template and line of code precipitated the error. That may be useful to see. I suspect it could be simply the line of your apps applciation.cfc or cfm that was enabling sessionmanagement, but it could be code that is referring to the session even after that, which would beg more questions.

Still, I realize that some getting this error might want to just restart the instance and get on with things. I'd regard that as more a work-around (giving up, due to a pragmatic need to "just get CF back up ASAP"). But if anyone wants to really solve things, I hope something in that may help to understand and really resolve the true root cause. And note again that much of what I share could be done and reviewed EVEN after one had to do such an urgent CF restart "to get it back up".

/Charlie (troubleshooter, carehart.org)

Report · Feb 20, 2018

Thanks for your observations, Charlie.

My last post is a summary of what I found about this issue. I studied the logs of over 40 instances in a period of 4 months. The instances are batch-started automatically as Windows services every morning. In that time the error occurred 16 times, sporadically, among the instances.

The requests involved are to CFCs, hence the use of onCFCRequest for preprocessing. From what we've seen, the exception occurs only for CFC requests. That is, requests of type http://.../myApp.cfc?method=func&arg=1. In addition, in every case, the error occurred within 10 minutes of startup. ColdFusion mentions coldfusion.runtime.CustomException because we use a custom throw.

The affected instance successfully runs some requests. But, as I said, it eventually gets overwhelmed by the exceptions, writing to the coldfusion-error.log and coldfusion-out.log by the minute.

We don't restart the instance, as you presume, as a workaround or "due to a pragmatic need". Its CPU use goes up and up towards 100% and the instance eventually hangs. Then the only way back up is a restart. Nevertheless, we continue to look for the root cause of the problem.

Report · Feb 21, 2018

OK, but I would be willing to be there's an explanation for the growing CPU that could be resolved, with the right use of the right tools (and a bit of time, perhaps less than an hour, though it would require recreating or waiting for the problem and being willing to dig into it when it happened, whether in the moment or via alerts/thread dumps). But as I said, I realize that someone may opt just to restart CF rather than try to dig into all that.

As for your finding the root of the session error to be some processing in oncfcrequest, that's certainly a useful clue, and I could imagine more that could be done to dig into that. But again I realize you may just want to leave this at what you have said.

And I'm happy to leave what I have said, in case someone down the road gets the same problem finds this, and maybe wants to dig in more. As the x-files saying went, "the answer is out there". 🙂

/Charlie (troubleshooter, carehart.org)

Report · Mar 05, 2018

i have traced the sequence of errors and exceptions down to a pair of lines in monitor.log similar to:

"Information","localhost-startStop-1","02/19/18","03:51:40",,"Starting Monitoring Server on port 56724."

"Error","localhost-startStop-1","02/19/18","03:52:01",,"Unable to establish loopback connection"

In ColdFusion 11 instances, the Monitoring Server always starts upon startup, even when, as in our c....

coldfusion.runtime.CustomException: Could not initialize class coldfusion.runtime.SessionTracker

1 Correct answer