Skip to main content
C
clixclix
Participant
December 22, 2020
Question

CF2018 sporadic Crash

  • December 22, 2020
  • 2 replies
  • 1035 views

Linux CentOS 7 - Apache 2.4

Sometimes our ColdFusion2018 service crashes without producing specific error logs.

How can we investigate further the causes? Thank you.

 

Here is the last error we found on coldfusion-error.log after last crash episode:

dic 19, 2020 2:48:33 PM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)
at org.apache.tomcat.util.net.NioChannel.write(NioChannel.java:140)
at org.apache.tomcat.util.net.NioBlockingSelector.write(NioBlockingSelector.java:101)
at org.apache.tomcat.util.net.NioSelectorPool.write(NioSelectorPool.java:152)
at org.apache.tomcat.util.net.NioEndpoint$NioSocketWrapper.doWrite(NioEndpoint.java:1261)
at org.apache.tomcat.util.net.SocketWrapperBase.doWrite(SocketWrapperBase.java:793)
at org.apache.tomcat.util.net.SocketWrapperBase.flushBlocking(SocketWrapperBase.java:746)
at org.apache.tomcat.util.net.SocketWrapperBase.flush(SocketWrapperBase.java:736)
at org.apache.coyote.ajp.AjpProcessor.putMessageArrayToOutputBuffer(AjpProcessor.java:1491)
at org.apache.coyote.ajp.AjpProcessor.getRealPathFromServer(AjpProcessor.java:1460)
at org.apache.coyote.ajp.AjpProcessor.getRealPath(AjpProcessor.java:1417)
at org.apache.catalina.core.StandardContext.getRealPath(StandardContext.java:4438)
at org.apache.catalina.core.ApplicationContext.getRealPath(ApplicationContext.java:390)
at org.apache.catalina.core.ApplicationContextFacade.getRealPath(ApplicationContextFacade.java:334)
at coldfusion.runtime.RealPathCache.getRealPath(RealPathCache.java:103)
at coldfusion.runtime.ServletContextWrapper.doGetRealPath(ServletContextWrapper.java:196)
at coldfusion.runtime.ServletContextWrapper._doGetRealPath(ServletContextWrapper.java:126)
at coldfusion.runtime.ServletContextWrapper.getRealPath(ServletContextWrapper.java:108)
at coldfusion.filter.FusionContext.getRealPath(FusionContext.java:1031)
at coldfusion.util.Utils.expandPath(Utils.java:503)
at coldfusion.runtime.CFPage.ExpandPath(CFPage.java:5720)
at cfincucina2ecfm2097799443._factor12(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1402)
at cfincucina2ecfm2097799443._factor19(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1241)
at cfincucina2ecfm2097799443._factor22(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1237)
at cfincucina2ecfm2097799443._factor24(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:715)
at cfincucina2ecfm2097799443.runPage(/opt/coldfusion2018/cfusion/wwwroot/hidden/incucina.cfm:1)
at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:262)
at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:735)
at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:565)
at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:597)
at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43)
at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
at coldfusion.filter.PathFilter.invoke(PathFilter.java:162)
at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)
at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:96)
at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:78)
at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60)
at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62)
at coldfusion.CfmServlet.service(CfmServlet.java:226)
at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46)
at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:490)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:357)
at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:422)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:853)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1587)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:834)

    This topic has been closed for replies.

    2 replies

    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    April 27, 2021

    The messages suggest the following to me

     

    1. "org.apache.coyote.ajp.AjpProcessor logError": an error in the connection between Tomcat (ColdFusion) and the web server.
    2. "SEVERE: null": the error involved a null application object, null request object or null response object.
    3. "java.io.IOException: Broken pipe": a socket disconnection between 2 servers.
    4. "java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)": this involves writing to a socket.

     

    Putting the above together, I would guess that a write-process involving the file incucina.cfm failed because the socket was closed or because connection was refused. The content to be written was probably null.

     

     

    P
    peterw1996572
    Participating Frequently
    April 26, 2021

    I'm getting the same issue (Centos7, Apache 2.4, CF2018)
    Have you managed to find anything more out?

    I'm getting it on both development and staging servers, we've not got this server into production yet.

    Charlie Arehart
    Community Expert
    Charlie ArehartCommunity Expert
    Community Expert
    April 26, 2021

    Peter (and clix), when you say "crash" do you mean you find the cf process no longer running? Or just not responsive? That's an important distinction.

     

    If it's truly down, unexpectedly, one place you can look for logs is the cfusion/bin folder. If the jvm crashed, you may find hs_err*.log files. (If you have multiple instances of CF, see the instancename/bin folder instead).

     

    Look also for any other logs in the cf logs folder, and pay attention specifically to the time it crashes or becomes unresponsive, as well as to when cf starts as there maybe a client there.

     

    Finally, if indeed cf is still running, but "unresponsive", it would be very helpful to have a cf monitor showing what's running. It may be that in fact many requests are hung, and once there are more than the "max simultaneous requests" value in the cf admin requests tuning page, cf queues any subsequent requests (for 60 seconds by default).

     

    As for monitoring, if you are on cf2018 or above, cf now includes the PMT (which many find challenging to get running for various reasons that can often be resolved), or a great 3rd party option is fusionreactor (which has  14 day free trial, but requires a cf restart to take effect). These tools will show WHAT (if any) requests are running, and a whole lot more that shines light into the cf black box. 

     

    I can help with installing/using these, as well as resolving this challenge you both face, if you prefer that over back-and-forth here. See my signature.

     

    Or write back with a response and I or others will carry on trying to help here. 

    /Charlie (troubleshooter, carehart. org)
    P
    peterw1996572
    Participating Frequently
    April 26, 2021

    I should have given more detail, I'm sometimes seeing a 500 error browser, in Chrome it shows as "500 err_empty_response".

     

    The browser tends to just reload itself immediately and often the page loads without a problem. It's not consistent either, I've not worked out how to replicate it, just sometimes a page will "crash", but refreshing (or allowing the browser to auto refresh) loads the page. The CF server itself remains up and running and the 2nd page load attempt is as quick as I'd expect.

    I get an error almost identical to clixclix's, from the coldfusion-out.log, starting with:

     

    Apr 22, 2021 10:54:23 AM org.apache.coyote.ajp.AjpProcessor logError
SEVERE: null
java.io.IOException: Broken pipe
        at java.base/sun.nio.ch.FileDispatcherImpl.write0(Native Method)
        at java.base/sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:47)
        at java.base/sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:113)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:79)
        at java.base/sun.nio.ch.IOUtil.write(IOUtil.java:50)
        at java.base/sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:466)

     

    I will see if I can find anything in an other logs and get FR installed on this new server

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices