• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
0

Coldfusion 11 random crash after 6-12 hours

Community Beginner ,
Jan 13, 2015 Jan 13, 2015

Copy link to clipboard

Copied

Hello,

We upgraded from Coldfusion 9 to Coldfusion 11.

System:

OS: Windows 2008 r2 64bit

Ram: 16 gb

Steps I took to do this upgrade.

1. uninstalled CF 9

2. uninstalled current version of java

3. reboot system

4. install latest Java 1.8.x

5. install Coldfusion 11

6. run coldfusion 11 updates

7. configured all datasources and settings in cfusion instance. (didn't want to import from 9)

8. test cfusion instance to make sure everything is working.

9. create are new instances for the websites. (4)

10. sign in to each instance and tweak settings.

11. assign 1 site to each instance for testing.

12. test the each site and make sure everything is still working.

13. follow Adobe lockdown guide as much as we can.

14 retest everything still seems fine.

run 6-12 hours and I get random crashes.

suddently get CF errors saying can't find includes which is a false error.

in the coldfusion-out.log I get this when the issue happens

Jan 13, 2015 8:04:20 AM org.apache.coyote.ajp.AjpProcessor getRealPathFromServer

SEVERE: Error in getRealPathFromConn

java.net.SocketException: Software caused connection abort: socket write error

  at java.net.SocketOutputStream.socketWrite0(Native Method)

  at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)

  at java.net.SocketOutputStream.write(SocketOutputStream.java:141)

  at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:403)

  at org.apache.coyote.ajp.AbstractAjpProcessor.getRealPath(AbstractAjpProcessor.java:1286)

  at org.apache.naming.resources.FileDirContext.doGetRealPath(FileDirContext.java:202)

  at org.apache.naming.resources.BaseDirContext.getRealPath(BaseDirContext.java:424)

  at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4660)

  at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:440)

  at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:333)

  at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:184)

  at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:116)

  at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:98)

  at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1142)

  at coldfusion.runtime.RuntimeServiceImpl.getRealPath(RuntimeServiceImpl.java:1156)

  at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:965)

  at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:937)

  at coldfusion.runtime.RuntimeServiceImpl.resolveTemplatePath(RuntimeServiceImpl.java:889)

  at coldfusion.tagext.lang.IncludeTag.setTemplate(IncludeTag.java:422)

  at cfsiteWideError2ecfm1972823533.runPage(D:\inetpub\cfadmin\JICustom\ErrorHandlers\siteWideError.cfm:6)

  at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:246)

  at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:736)

  at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:572)

  at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)

  at coldfusion.filter.CfincludeFilter.include(CfincludeFilter.java:33)

  at coldfusion.filter.ExceptionFilter.runSiteWideHandler(ExceptionFilter.java:626)

  at coldfusion.filter.ExceptionFilter.handleException(ExceptionFilter.java:311)

  at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:108)

  at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)

  at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)

  at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:58)

  at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)

  at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)

  at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)

  at coldfusion.CfmServlet.service(CfmServlet.java:219)

  at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)

  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)

  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

  at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)

  at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)

  at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)

  at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)

  at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:220)

  at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)

  at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:501)

  at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)

  at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

  at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)

  at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:422)

  at org.apache.coyote.ajp.AjpProcessor.process(AjpProcessor.java:199)

  at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:607)

  at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:314)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

  at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

  at java.lang.Thread.run(Thread.java:745)

in windows event log I get

Faulting application name: w3wp.exe, version: 7.5.7601.17514, time stamp: 0x4ce7afa2

Faulting module name: isapi_redirect.dll, version: 1.2.40.0, time stamp: 0x545a1577

Exception code: 0xc0000005

Fault offset: 0x00000000000158a7

Faulting process id: 0x1204

Faulting application start time: 0x01d02e62f49ae158

Faulting application path: c:\windows\system32\inetsrv\w3wp.exe

Faulting module path: D:\applications\ColdFusion11\config\wsconfig\9\isapi_redirect.dll

Report Id: 53bd00cb-9a7c-11e4-9717-0026553456ec

after getting these errors and doing a google search I saw the connector tuning steps.

so I tried that but still get the same crashes.

I don't know what to look at next, I'm at a total loss on this issue since everything ran fine in CF 9.

Thank you in advance for any help and if you need more information let me know.

Views

2.4K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines

correct answers 1 Correct answer

Guide , Jan 15, 2015 Jan 15, 2015

Capture.JPGNot sure what that isapi log error means off hand. Perhaps some searching.

A couple other tomcat connector tuning items I have noticed help.
-Initial setting for pool threads otherwise the default value when not specified of 10 can be small.
-Some people have noticed performance, stability improvements by using AJP-APR instead of AJP-BIO.

EG
worker.list=FLW
worker.FLW.type=ajp13
worker.FLW.host=localhost
worker.FLW.port=8013
worker.FLW.max_reuse_connections=270
worker.FLW.connection_pool_size=1000
worker.

...

Votes

Translate

Translate
Guide ,
Jan 13, 2015 Jan 13, 2015

Copy link to clipboard

Copied

Hello,

Do any errors or warnings appear in the "CF IIS" connector log?

\ColdFusion11\config\wsconfig\N\isapi_redirect.log

You say CF11 is updated, updater3 being current. Since you connected CF to IIS sites after updater3 likely
the post updater3 step to manually increment isapi_redirect.dll has occurred. Case that is not true what date size stamp does dll show EG 10 November 2014 401,920 bytes.

CF11 is using Java 1.8.x? CFadmin i system information.


Tomcat tunning, recommended in many places, involves changes to workers.properties and server.xml I expect you have altered both files and restarted IIS CF to apply.

Perhaps you could do well to monitor CF Java and tomcat with Jconsole. Since you have JDK 1.8.x Java mission control JMC is present and  useful to monitor java and tomcat.

HTH, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 14, 2015 Jan 14, 2015

Copy link to clipboard

Copied

With CF11 updater3, isapi_redirect.dll properties file version 1.2.40.0, would be an easier way to know the connector is right.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 15, 2015 Jan 15, 2015

Copy link to clipboard

Copied

Carl sorry for the slow response, I was getting a few logs you asked for.

isapi_redirect.dll is version 1.2.40.0 and date and size exactly what you typed.

isapi_redirect.log:

[Thu Jan 15 04:34:02.054 2015] [5960:4632] [error] HttpExtensionProc::jk_isapi_plugin.c (2622): failed to init service for request.

[Thu Jan 15 04:37:35.515 2015] [5960:4632] [error] HttpExtensionProc::jk_isapi_plugin.c (2622): failed to init service for request.

in jconsole is there something specific I should look at. just want to make sure I'm looking at the correct thing.

yes I switched it to Java 1.8, and copied the tools.jar file to each instance

Connector for our site - the CF instance has 8 sites total in it but this one is our higher traffic one.

worker.list=FLW

worker.FLW.type=ajp13

worker.FLW.host=localhost

worker.FLW.port=8013

worker.FLW.max_reuse_connections=270

worker.FLW.connection_pool_size=1000

worker.FLW.connection_pool_timeout=60

server.xml

<Connector port="8013" protocol="AJP/1.3" redirectPort="8446" tomcatAuthentication="false" maxThreads="3800" connectionTimeout="60000"/>

of course the max threads is sum of the 8 sites.

anything else you might need let me know.. would love to track the issue down.

Thanks for all your help.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 15, 2015 Jan 15, 2015

Copy link to clipboard

Copied

Capture.JPGNot sure what that isapi log error means off hand. Perhaps some searching.

A couple other tomcat connector tuning items I have noticed help.
-Initial setting for pool threads otherwise the default value when not specified of 10 can be small.
-Some people have noticed performance, stability improvements by using AJP-APR instead of AJP-BIO.

EG
worker.list=FLW
worker.FLW.type=ajp13
worker.FLW.host=localhost
worker.FLW.port=8013
worker.FLW.max_reuse_connections=270
worker.FLW.connection_pool_size=1000
worker.FLW.connection_pool_minsize=100
worker.FLW.connection_pool_timeout=60


server.xml
<Connector port="8013" protocol="AJP/1.3" redirectPort="8446" tomcatAuthentication="false" maxThreads="3800" minSpareThreads="100" connectionTimeout="60000"/>

Note CF and IIS need restart to re-read properties and xml change.

AJP-APR download tomcat native, extract and place x64 dll in CF11\instance\lib. CF will need a restart to apply that.

Java 8 note. Did you replace msvcr100.dll in CF11\instance\bin with the one from JDK8\bin? Perhaps wise to take copy of existing file before so you can roll back. CF requires stop to apply.


Attached a Jconsole snippy. Notice AJP APR not BIO since tomcat native applied.


HTH again, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 15, 2015 Jan 15, 2015

Copy link to clipboard

Copied

Carl thanks I will try the things you mentioned and let you know.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jan 19, 2015 Jan 19, 2015

Copy link to clipboard

Copied

@Tmontour, how did things work out for you?

BTW, if still troublesome, then here’s a question: when you report above that you confirmed the date of “the isapi_redirect.dll” (singular), I see that you refer in one spot to a config/wsconfig/9/ folder. So I suspect you may have more than one such folder (with numbers greater than or lower than 9). Are you confirming that you checked the dates of ALL of those? And that these tuning tweaks are in the workers.properties for each of them (at a minimum the idle thread timeout one)?

/charlie


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 20, 2015 Jan 20, 2015

Copy link to clipboard

Copied

@carl, I added the modifications you talked about but it still became unresponsive with the same errors as above.

I ended up rolling back 2 of our servers to CF 9 since everything works in that version. I will keep testing on our other 2 servers and see if I can come up with something. going to try and split the site that uses connector 9 to its own application pool in IIS.

I also attached a screen grab of jconsole when the crash was happening.

.jconsole_reading-at-crash2.png

@ Charlie, I did check all the folders under wsconfig and they all have the same dll's, I also modified all the connectors as we have 34 of them in total.

Thanks

Troy

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 20, 2015 Jan 20, 2015

Copy link to clipboard

Copied

Hi Troy,

Sorry to hear that is still not working well. Very interesting for me, painful for you.

Curious the diagrams you snip show two very different usage patterns of the tomcat AJP. Values for
connectioncount, currentthreadcount, currentthreadsbusy and keepalivecount vary greatly. Was system ok at 10:00 then failed at 10:02 or are those charts over different days or sites?

JMeter might be handy to simulate load since your production payload has moved back to CF9.

Regards, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 20, 2015 Jan 20, 2015

Copy link to clipboard

Copied

PS

Since your currentthreadcount is > 100 perhaps do well to increase the pool / thread minimum or initial setting beyond 100 that I specified earlier. Tomcat reminders - minimum or initial 10 by default when not
specified. Now have seen some typical load usage, tomcat reading seems to be a good idea to set minimum or initial to whatever is needed to handle your normal usage. So from jconsole observation double that 100 to 200 in server.xml and workers.properties, both CF and IIS restart to effect change.

While Jconsole was running did memory or cpu show some spikes? Much time taken in Full garbage collection?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 01, 2015 Apr 01, 2015

Copy link to clipboard

Copied

LATEST

Carl,

I wanted to make sure I sent a reply to this thread. Anit from Adobe helped us do some testing and fixing, so I now have a stable environment.

there was a bug in the connector that was causing our crash which was fixed in updater 4.

Thank you so much for your help as it pointed us in the right direction.

Thanks again

Troy

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 21, 2015 Jan 21, 2015

Copy link to clipboard

Copied

Carl,

to answer your question on the diagrams it was the same day roughly 8 hour window and yes at 10:00 am it was fine then 10:02 failed.

I will make the other tweaks to server.xml and workers.properties and let you know. might be after this weekend since we have an event starting tomorrow and I don't want to cause any other issues during it.

Memory,cpu all looked fine for our application- cpu was around 40-60%, memory was sitting at 9gb used and 5gb free which is normal.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jan 21, 2015 Jan 21, 2015

Copy link to clipboard

Copied

oops Garbage Collector seemed good when I looked at it. I don't remember the time in between each collection but it never seemed to be maxed out in the heap.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jan 21, 2015 Jan 21, 2015

Copy link to clipboard

Copied

@Troy, I wonder if you may be barking up the wrong tree...or looking down the wrong rabbit hole. Pick your favorite analogy. 🙂

I responded with my question because you made reference to the connector, and thank for confirming you're at least not suffering some of the common problems folks (unexpectedly) have with that. And while I do realize you pointed to an error in the event log that showed the connector crashed, and Carl reasonably followed you down that hole, I wonder if your problem may have nothing to do with the connector after all.

First of all, to be clear, if CF requests pile up in CF for some reason, that would "backup" into the connector and could cause problems like that, just as a blogged sewer pipe can backup into your basement, but the problem may be a root in the pipe 30 feet from your home, not in your basement where you "experience" the problem. 🙂

And second, I see your original note you say that some cfinclude calls started failing. Well that too would have absolutely nothing to do with the connector (again, I was focused on the bulk of what was being discussed and fell into wondering about the connector).

So have you looked at the CF11/cfusion/logs directory (or instancename instance of cfusion, if you are running CF Enterprise and have multiple instances)? Look especially at the coldfusion-out.log and coldfusion-error.log files, and find the time that you experienced the problem. Or look at least at the logs before you last restarted CF (after having the problem, even if that may have been minutes or hours later). Look through those logs for the minutes and hours prior to the restart, to see if you see anything that may be unusual or a clearly marked error.

You could have any of a number of possible problems in CF that may have led you to "suddently get CF errors saying can't find includes which is a false error." For instance, you could have outofmemory conditions (in the heap or permgen, or referring to swapspace--which generally means CF couldn't get from the OS the memory you told it it could), and you may also find logs at the time of an outage in the coldfusion11/cfusion/bin folder (believe it or not), which are where certain JVM crashes are logged.

So I'm just saying maybe the problem is not the connector at all. I realize you have "moved back" to CF9, but if you've not uninstalled CF11 you may at least be able to consider the logs. I help folks find and resolve such problems every day, so it's not as impossible a situation as it may have seemed. While some can seem daunting and impossible (or google searches may be fruitless, overwhelming, or misleading), the good news is that nearly every problem can be solved, and some quickly, once you look in the right places and connect the dots. Some of us are just more adept at doing that as we do it daily.

Trying to resolve things over email or forums can often be quite challenging (and sometimes there's no substitute for having someone "look over your shoulders" with you remotely, and there are some of us who do that, helping solve CF server problems for a living, which you may turn to), but we're here to try to help if we can.


/Charlie (troubleshooter, carehart.org)

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jan 24, 2015 Jan 24, 2015

Copy link to clipboard

Copied

Tmontour wrote:

Hello,

We upgraded from Coldfusion 9 to Coldfusion 11.

System:

OS: Windows 2008 r2 64bit

Ram: 16 gb

Steps I took to do this upgrade.

1. uninstalled CF 9

2. uninstalled current version of java

3. reboot system

4. install latest Java 1.8.x

5. install Coldfusion 11

6. run coldfusion 11 updates

...

May we assume from 'updates' that you applied more than one? If so, then the ColdFusion installation may be unclean.

You should completely re-install ColdFusion and apply just Update 3. This update is cumulative, and so includes previous Updates.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Jan 26, 2015 Jan 26, 2015

Copy link to clipboard

Copied

Hi Folks, Considering the topic of clean CF installs, the latest refresh of CF11 installer has update 3 built in as well as Java 8 1.8.0_25.

Regards, Carl.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Jan 26, 2015 Jan 26, 2015

Copy link to clipboard

Copied

Thanks, Carl. Handy.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Resources
Documentation