CF2018 sporadic Crash

New Here ,
Dec 22, 2020 Dec 22, 2020

Copy link to clipboard

Copied

Linux CentOS 7 - Apache 2.4

Sometimes our ColdFusion2018 service crashes without producing specific error logs.

How can we investigate further the causes? Thank you.

 

Here is the last error we found on coldfusion-error.log after last crash episode:

dic 19, 2020 2:48:33 PM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:140)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1261)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:793)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:746)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:736)
at org.apache.coyote.ajp.AjpProcessor.putMessageArrayToOutputBuffer(AjpProcessor.java:1491)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:1460)
at org.apache.coyote.ajp.AjpProcessor.getRealPath(AjpProcessor.java:1417)
at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4438)
at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:390)
at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:334)
at coldfusion.runtime.RealPathCache.getRealPath(RealPathCache.java:103)
at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:196)
at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:126)
at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:108)
at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:1031)
at coldfusion.util.Utils.expandPath(Utils.java:503)
at coldfusion.runtime.CFPage.ExpandPath(CFPage.java:5720)
at cfincucina2ecfm2097799443._factor12(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1402)
at cfincucina2ecfm2097799443._factor19(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1241)
at cfincucina2ecfm2097799443._factor22(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1237)
at cfincucina2ecfm2097799443._factor24(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:715)
at cfincucina2ecfm2097799443.runPage(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:262)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:735)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:565)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:597)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:96)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:78)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:226)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:422)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)

Views

125

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 26, 2021 Apr 26, 2021

Copy link to clipboard

Copied

I'm getting the same issue (Centos7, Apache 2.4, CF2018)
Have you managed to find anything more out?

I'm getting it on both development and staging servers, we've not got this server into production yet.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 26, 2021 Apr 26, 2021

Copy link to clipboard

Copied

Peter (and clix), when you say "crash" do you mean you find the cf process no longer running? Or just not responsive? That's an important distinction.

 

If it's truly down, unexpectedly, one place you can look for logs is the cfusion/bin folder. If the jvm crashed, you may find hs_err*.log files. (If you have multiple instances of CF, see the instancename/bin folder instead).

 

Look also for any other logs in the cf logs folder, and pay attention specifically to the time it crashes or becomes unresponsive, as well as to when cf starts as there maybe a client there.

 

Finally, if indeed cf is still running, but "unresponsive", it would be very helpful to have a cf monitor showing what's running. It may be that in fact many requests are hung, and once there are more than the "max simultaneous requests" value in the cf admin requests tuning page, cf queues any subsequent requests (for 60 seconds by default).

 

As for monitoring, if you are on cf2018 or above, cf now includes the PMT (which many find challenging to get running for various reasons that can often be resolved), or a great 3rd party option is fusionreactor (which has  14 day free trial, but requires a cf restart to take effect). These tools will show WHAT (if any) requests are running, and a whole lot more that shines light into the cf black box. 

 

I can help with installing/using these, as well as resolving this challenge you both face, if you prefer that over back-and-forth here. See my signature.

 

Or write back with a response and I or others will carry on trying to help here. 


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Apr 26, 2021 Apr 26, 2021

Copy link to clipboard

Copied

I should have given more detail, I'm sometimes seeing a 500 error browser, in Chrome it shows as "500 err_empty_response".

 

The browser tends to just reload itself immediately and often the page loads without a problem. It's not consistent either, I've not worked out how to replicate it, just sometimes a page will "crash", but refreshing (or allowing the browser to auto refresh) loads the page. The CF server itself remains up and running and the 2nd page load attempt is as quick as I'd expect.

I get an error almost identical to clixclix's, from the coldfusion-out.log, starting with:

 

Apr 22, 2021 10:54:23 AM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)

 

I will see if I can find anything in an other logs and get FR installed on this new server

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Jun 09, 2021 Jun 09, 2021

Copy link to clipboard

Copied

LATEST

Having looked in the wsconfig/1/mod_jk.log, I was led to this post and changing the `workers.properties` value of `heartbeat_interval` has fixed this for me:
https://coldfusion.adobe.com/2019/02/apache-crashes-serving-multiple-thread-requests-using-cf2018/

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 27, 2021 Apr 27, 2021

Copy link to clipboard

Copied

The messages suggest the following to me

 

  1. "org.apache.coyote.ajp.AjpProcessor logError": an error in the connection between Tomcat (ColdFusion) and the web server.
  2. "SEVERE: null": the error involved a null application object, null request object or null response object.
  3. "java.io.IOException: Broken pipe": a socket disconnection between 2 servers.
  4. "java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)": this involves writing to a socket.

 

Putting the above together, I would guess that a write-process involving the file incucina.cfm failed because the socket was closed or because connection was refused. The content to be written was probably null.

 

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines