CF21 - java.lang.ClassNotFoundException: net.sf.jasperreports.engine.xml.JasperDesignFactory
Hi all -
I have a clustered environment of three CF Enterprise servers (bare metal) each running four clustered instances. After much testing, we recently put these into production and are experiencing frequent errors. This seemingly causes the instance to hang, and requires manually stopping the instance, waiting a few moments and then restarting. This is happening at random to all instances, across all three bare metal servers, multiple times a day. It's a full time job identifying the failed server and CF instance, and then restarting it. Does anyone have any suggestions on how to diagnose the issue?
From the coldfusion-error log on a failed instance, we see:
Jul 19, 2021 6:27:59 PM org.apache.commons.digester3.Digester startElement SEVERE: Begin event threw exception java.lang.ClassNotFoundException: net.sf.jasperreports.engine.xml.JasperDesignFactory at coldfusion.bootstrap.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:303) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) at org.apache.commons.digester3.FactoryCreateRule.getFactory(FactoryCreateRule.java:379) at org.apache.commons.digester3.FactoryCreateRule.begin(FactoryCreateRule.java:276) at org.apache.commons.digester3.Digester.startElement(Digester.java:1350) at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) at org.apache.xerces.impl.dtd.XMLDTDValidator.startElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl$ContentDispatcher.scanRootElementHook(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester3.Digester.parse(Digester.java:1642) at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:240) at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:227) at net.sf.jasperreports.engine.xml.JRXmlLoader.load(JRXmlLoader.java:215) at coldfusion.runtime.report.Report.compileXml(Report.java:827) at coldfusion.runtime.report.Report.access$200(Report.java:90) at coldfusion.runtime.report.Report$3.fetch(Report.java:860) at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:153) at coldfusion.util.SoftCache.get(SoftCache.java:92) at coldfusion.runtime.report.Report._compileReport(Report.java:276) at coldfusion.runtime.report.Report.compileReport(Report.java:176) at coldfusion.runtime.report.Report.runReport(Report.java:425) at coldfusion.tagext.report.ReportTag.doEndTag(ReportTag.java:626) at cfvoucher2ecfm302681485._factor4(/var/www/html/services/vouchers/voucher.cfm:602) at cfvoucher2ecfm302681485._factor28(/var/www/html/services/vouchers/voucher.cfm:599) at cfvoucher2ecfm302681485._factor34(/var/www/html/services/vouchers/voucher.cfm:46) at cfvoucher2ecfm302681485.runPage(/var/www/html/services/vouchers/voucher.cfm:1) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:257) at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:749) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:578) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:572) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:162) at coldfusion.filter.IpFilter.invoke(IpFilter.java:45) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97) at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:231) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:330) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:459) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Jul 19, 2021 6:27:59 PM org.apache.commons.digester3.Digester parse SEVERE: An error occurred while parsing XML from '(already loaded from stream)', see nested exceptions org.xml.sax.SAXParseException; lineNumber: 4; columnNumber: 236; Error at line 4 char 236: net.sf.jasperreports.engine.xml.JasperDesignFactory at org.apache.commons.digester3.Digester.createSAXException(Digester.java:3118) at org.apache.commons.digester3.Digester.createSAXException(Digester.java:3146) at org.apache.commons.digester3.Digester.startElement(Digester.java:1355) at org.apache.xerces.parsers.AbstractSAXParser.startElement(Unknown Source) at org.apache.xerces.impl.dtd.XMLDTDValidator.startElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanStartElement(Unknown Source) at org.apache.xerces.impl.XMLDocumentScannerImpl$ContentDispatcher.scanRootElementHook(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl$FragmentContentDispatcher.dispatch(Unknown Source) at org.apache.xerces.impl.XMLDocumentFragmentScannerImpl.scanDocument(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XML11Configuration.parse(Unknown Source) at org.apache.xerces.parsers.XMLParser.parse(Unknown Source) at org.apache.xerces.parsers.AbstractSAXParser.parse(Unknown Source) at org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser.parse(Unknown Source) at org.apache.commons.digester3.Digester.parse(Digester.java:1642) at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:240) at net.sf.jasperreports.engine.xml.JRXmlLoader.loadXML(JRXmlLoader.java:227) at net.sf.jasperreports.engine.xml.JRXmlLoader.load(JRXmlLoader.java:215) at coldfusion.runtime.report.Report.compileXml(Report.java:827) at coldfusion.runtime.report.Report.access$200(Report.java:90) at coldfusion.runtime.report.Report$3.fetch(Report.java:860) at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:153) at coldfusion.util.SoftCache.get(SoftCache.java:92) at coldfusion.runtime.report.Report._compileReport(Report.java:276) at coldfusion.runtime.report.Report.compileReport(Report.java:176) at coldfusion.runtime.report.Report.runReport(Report.java:425) at coldfusion.tagext.report.ReportTag.doEndTag(ReportTag.java:626) at cfvoucher2ecfm302681485._factor4(/var/www/html/services/vouchers/voucher.cfm:602) at cfvoucher2ecfm302681485._factor28(/var/www/html/services/vouchers/voucher.cfm:599) at cfvoucher2ecfm302681485._factor34(/var/www/html/services/vouchers/voucher.cfm:46) at cfvoucher2ecfm302681485.runPage(/var/www/html/services/vouchers/voucher.cfm:1) at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:257) at coldfusion.tagext.lang.IncludeTag.handlePageInvoke(IncludeTag.java:749) at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:578) at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65) at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:572) at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:43) at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40) at coldfusion.filter.PathFilter.invoke(PathFilter.java:162) at coldfusion.filter.IpFilter.invoke(IpFilter.java:45) at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:97) at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:81) at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28) at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38) at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:60) at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38) at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22) at coldfusion.filter.CachingFilter.invoke(CachingFilter.java:62) at coldfusion.CfmServlet.service(CfmServlet.java:231) at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:311) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:46) at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:47) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:97) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:542) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:143) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:78) at org.apache.catalina.ha.session.JvmRouteBinderValve.invoke(JvmRouteBinderValve.java:182) at org.apache.catalina.ha.tcp.ReplicationValve.invoke(ReplicationValve.java:330) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:359) at org.apache.coyote.ajp.AjpProcessor.service(AjpProcessor.java:459) at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:888) at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1597) at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) at java.base/java.lang.Thread.run(Thread.java:834) Caused by: java.lang.ClassNotFoundException: net.sf.jasperreports.engine.xml.JasperDesignFactory at coldfusion.bootstrap.BootstrapClassLoader.loadClass(BootstrapClassLoader.java:303) at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521) at org.apache.commons.digester3.FactoryCreateRule.getFactory(FactoryCreateRule.java:379) at org.apache.commons.digester3.FactoryCreateRule.begin(FactoryCreateRule.java:276) at org.apache.commons.digester3.Digester.startElement(Digester.java:1350) ... 74 more
Thanks for any suggestions,
-Tony
