Copy link to clipboard
Copied
Linux CentOS 7 - Apache 2.4
Sometimes our ColdFusion2018 service crashes without producing specific error logs.
How can we investigate further the causes? Thank you.
Here is the last error we found on coldfusion-error.log after last crash episode:
dic 19, 2020 2:48:33 PM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:140)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1261)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:793)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:746)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:736)
at org.apache.coyote.ajp.AjpProcessor.putMessageArrayToOutputBuffer(AjpProcessor.java:1491)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:1460)
at org.apache.coyote.ajp.AjpProcessor.getRealPath(AjpProcessor.java:1417)
at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4438)
at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:390)
at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:334)
at coldfusion.runtime.RealPathCache.getRealPath(RealPathCache.java:103)
at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:196)
at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:126)
at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:108)
at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:1031)
at coldfusion.util.Utils.expandPath(Utils.java:503)
at coldfusion.runtime.CFPage.ExpandPath(CFPage.java:5720)
at cfincucina2ecfm2097799443._factor12(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1402)
at cfincucina2ecfm2097799443._factor19(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1241)
at cfincucina2ecfm2097799443._factor22(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1237)
at cfincucina2ecfm2097799443._factor24(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:715)
at cfincucina2ecfm2097799443.runPage(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:262)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:735)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:565)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:597)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:96)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:78)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:226)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:422)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)
Copy link to clipboard
Copied
I'm getting the same issue (Centos7, Apache 2.4, CF2018)
Have you managed to find anything more out?
I'm getting it on both development and staging servers, we've not got this server into production yet.
Copy link to clipboard
Copied
Peter (and clix), when you say "crash" do you mean you find the cf process no longer running? Or just not responsive? That's an important distinction.
If it's truly down, unexpectedly, one place you can look for logs is the cfusion/bin folder. If the jvm crashed, you may find hs_err*.log files. (If you have multiple instances of CF, see the instancename/bin folder instead).
Look also for any other logs in the cf logs folder, and pay attention specifically to the time it crashes or becomes unresponsive, as well as to when cf starts as there maybe a client there.
Finally, if indeed cf is still running, but "unresponsive", it would be very helpful to have a cf monitor showing what's running. It may be that in fact many requests are hung, and once there are more than the "max simultaneous requests" value in the cf admin requests tuning page, cf queues any subsequent requests (for 60 seconds by default).
As for monitoring, if you are on cf2018 or above, cf now includes the PMT (which many find challenging to get running for various reasons that can often be resolved), or a great 3rd party option is fusionreactor (which has 14 day free trial, but requires a cf restart to take effect). These tools will show WHAT (if any) requests are running, and a whole lot more that shines light into the cf black box.
I can help with installing/using these, as well as resolving this challenge you both face, if you prefer that over back-and-forth here. See my signature.
Or write back with a response and I or others will carry on trying to help here.
Copy link to clipboard
Copied
I should have given more detail, I'm sometimes seeing a 500 error browser, in Chrome it shows as "500 err_empty_response".
The browser tends to just reload itself immediately and often the page loads without a problem. It's not consistent either, I've not worked out how to replicate it, just sometimes a page will "crash", but refreshing (or allowing the browser to auto refresh) loads the page. The CF server itself remains up and running and the 2nd page load attempt is as quick as I'd expect.
I get an error almost identical to clixclix's, from the coldfusion-out.log, starting with:
Apr 22, 2021 10:54:23 AM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
I will see if I can find anything in an other logs and get FR installed on this new server
Copy link to clipboard
Copied
Having looked in the wsconfig/1/mod_jk.log, I was led to this post and changing the `workers.properties` value of `heartbeat_interval` has fixed this for me:
https://coldfusion.adobe.com/2019/02/apache-crashes-serving-multiple-thread-requests-using-cf2018/
Copy link to clipboard
Copied
The messages suggest the following to me
Putting the above together, I would guess that a write-process involving the file incucina.cfm failed because the socket was closed or because connection was refused. The content to be written was probably null.