Skip to main content
G
GuitsBoy
Inspiring
February 14, 2011
Question

Troubleshooting CF8 + CentOS5 lockups

  • February 14, 2011
  • 1 reply
  • 5281 views

Hi guys, I was hoping you might be able to help me troubleshoot this issue I'm having.  We have 5 identical servers running our application.  They are all configured identically.  About a week ago, web1, our first node, started locking up a few times a day, with the coldfusion process jacked up at 100% cup usage.  I have verified the settings are identical to teh other 4 boxes, and I have copied over a fresh copy of the codebase.  I have also applied all updates on the misbehaving box, as well as updateing to CF8.01 hotfix 4, and JRE 1.6.0_23.  I have cleared all cache, and I have rebooted the machine several times.  The hangups are still happening, seemingly at random.

Here's the specs on the machines:

Dell 1950's, Two dual core 3.0 GHz xeon's, 8GB memory, two 147GB 15k scsi in raid 1.

Cent OS 5.5 64 Bit

Coldfusion 8.01 Standard (32 bit)

Apache 2.2.3 (32 bit)

Here are some log file snippets from when the machine is experiencing problems:

CF application log

"Error","jrpp-40","02/10/11","09:31:51","SO","GC overhead limit exceeded The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
"Error","jrpp-35","02/10/11","09:31:51",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
"Error","jrpp-31","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
"Error","jrpp-30","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 61 "
"Error","jrpp-9","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
"Error","jrpp-51","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 61 "
"Error","jrpp-49","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
"Error","jrpp-11","02/10/11","09:35:21","SO","GC overhead limit exceeded The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
"Error","jrpp-21","02/10/11","09:50:14","SO","The request has exceeded the allowable time limit Tag: CFQUERY The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 247 "

CF Exception log

"Error","jrpp-40","02/10/11","09:31:51","SO","GC overhead limit exceeded The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.lang.AbstractStringBuilder.<init>(AbstractStringBuilder.java:45)
        at java.lang.StringBuilder.<init>(StringBuilder.java:68)
        at java.io.UnixFileSystem.resolve(UnixFileSystem.java:93)
        at java.io.File.<init>(File.java:207)
        at java.io.File.listFiles(File.java:1056)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:54)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:62)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:48)
        at coldfusion.compiler.CustomTagTable.<init>(CustomTagTable.java:34)
        at coldfusion.runtime.RuntimeServiceImpl.resolveTemplateName(RuntimeServiceImpl.java:579)
        at coldfusion.tagext.lang.ModuleTag.setName(ModuleTag.java:84)
        at coldfusion.tagext.lang.CustomTag.setName(CustomTag.java:34)
        at cfdsp_edit2ecfm45629791._factor5(/san/html/sourceone/admin/dsp_edit.cfm:39)
        at cfdsp_edit2ecfm45629791._factor14(/san/html/sourceone/admin/dsp_edit.cfm:37)
        at cfdsp_edit2ecfm45629791.runPage(/san/html/sourceone/admin/dsp_edit.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Switch2ecfm1533486892._factor41(/san/html/sourceone/admin/fbx_Switch.cfm:346)
        at cffbx_Switch2ecfm1533486892._factor109(/san/html/sourceone/admin/fbx_Switch.cfm:340)
        at cffbx_Switch2ecfm1533486892.runPage(/san/html/sourceone/admin/fbx_Switch.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor4(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:242)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor5(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667.runPage(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1792679210.runPage(/san/html/sourceone/index.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
"Error","jrpp-35","02/10/11","09:31:51",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cfapp_GlobalRequests2ecfm789074969.runPage(/san/html/dynamic/app_GlobalRequests.cfm:18)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfapp_Globals2ecfm2092923734.runPage(/san/html/dynamic/app_Globals.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:79)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-31","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cfapp_GlobalRequests2ecfm789074969.runPage(/san/html/dynamic/app_GlobalRequests.cfm:18)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfapp_Globals2ecfm2092923734.runPage(/san/html/dynamic/app_Globals.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:79)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.filter.RequestThrottleFilter.invoke(RequestThrottleFilter.java:126)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-30","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 61 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:61)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-9","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cfapp_GlobalRequests2ecfm789074969.runPage(/san/html/dynamic/app_GlobalRequests.cfm:18)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfapp_Globals2ecfm2092923734.runPage(/san/html/dynamic/app_Globals.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:79)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-51","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 61 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:61)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-49","02/10/11","09:31:52",,"'' The specific sequence of files included or processed is: /san/html/dynamic/index.cfm, line: 18 "
java.lang.NullPointerException
        at coldfusion.util.Utils.getDirectoryFromPath(Utils.java:294)
        at coldfusion.runtime.CFPage.GetDirectoryFromPath(CFPage.java:2909)
        at cfapp_GlobalRequests2ecfm789074969.runPage(/san/html/dynamic/app_GlobalRequests.cfm:18)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfapp_Globals2ecfm2092923734.runPage(/san/html/dynamic/app_Globals.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Settings2ecfm2009881526.runPage(/san/html/dynamic/fbx_Settings.cfm:79)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF502ecfm1499241541._factor3(/san/html/dynamic/fbx_fusebox30_CF50.cfm:159)
        at cffbx_fusebox30_CF502ecfm1499241541._factor6(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at cffbx_fusebox30_CF502ecfm1499241541.runPage(/san/html/dynamic/fbx_fusebox30_CF50.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1943790857.runPage(/san/html/dynamic/index.cfm:20)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
"Error","jrpp-11","02/10/11","09:35:21","SO","GC overhead limit exceeded The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 39 "
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.io.File.listFiles(File.java:1056)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:54)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:62)
        at coldfusion.compiler.CustomTagTable.scanTags(CustomTagTable.java:48)
        at coldfusion.compiler.CustomTagTable.<init>(CustomTagTable.java:34)
        at coldfusion.runtime.RuntimeServiceImpl.resolveTemplateName(RuntimeServiceImpl.java:579)
        at coldfusion.tagext.lang.ModuleTag.setName(ModuleTag.java:84)
        at coldfusion.tagext.lang.CustomTag.setName(CustomTag.java:34)
        at cfdsp_edit2ecfm45629791._factor5(/san/html/sourceone/admin/dsp_edit.cfm:39)
        at cfdsp_edit2ecfm45629791._factor14(/san/html/sourceone/admin/dsp_edit.cfm:37)
        at cfdsp_edit2ecfm45629791.runPage(/san/html/sourceone/admin/dsp_edit.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_Switch2ecfm1533486892._factor41(/san/html/sourceone/admin/fbx_Switch.cfm:346)
        at cffbx_Switch2ecfm1533486892._factor109(/san/html/sourceone/admin/fbx_Switch.cfm:340)
        at cffbx_Switch2ecfm1533486892.runPage(/san/html/sourceone/admin/fbx_Switch.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor4(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:242)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor5(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667.runPage(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1792679210.runPage(/san/html/sourceone/index.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
"Error","jrpp-21","02/10/11","09:50:14","SO","The request has exceeded the allowable time limit Tag: CFQUERY The specific sequence of files included or processed is: /san/html/sourceone/index.cfm, line: 247 "
coldfusion.runtime.CustomException: The request has exceeded the allowable time limit Tag: CFQUERY
        at coldfusion.tagext.lang.ThrowTag.doStartTag(ThrowTag.java:124)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor4(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:247)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667._factor5(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at cffbx_fusebox30_CF50_nix2ecfm1062751667.runPage(/san/html/sourceone/fbx_fusebox30_CF50_nix.cfm:1)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:2661)
        at cfindex2ecfm1792679210.runPage(/san/html/sourceone/index.cfm:11)
        at coldfusion.runtime.CfJspPage.invoke(CfJspPage.java:196)
        at coldfusion.tagext.lang.IncludeTag.doStartTag(IncludeTag.java:370)
        at coldfusion.filter.CfincludeFilter.invoke(CfincludeFilter.java:65)
        at coldfusion.filter.ApplicationFilter.invoke(ApplicationFilter.java:279)
        at coldfusion.filter.RequestMonitorFilter.invoke(RequestMonitorFilter.java:48)
        at coldfusion.filter.MonitoringFilter.invoke(MonitoringFilter.java:40)
        at coldfusion.filter.PathFilter.invoke(PathFilter.java:86)
        at coldfusion.filter.ExceptionFilter.invoke(ExceptionFilter.java:70)
        at coldfusion.filter.BrowserDebugFilter.invoke(BrowserDebugFilter.java:74)
        at coldfusion.filter.ClientScopePersistenceFilter.invoke(ClientScopePersistenceFilter.java:28)
        at coldfusion.filter.BrowserFilter.invoke(BrowserFilter.java:38)
        at coldfusion.filter.NoCacheFilter.invoke(NoCacheFilter.java:46)
        at coldfusion.filter.GlobalsFilter.invoke(GlobalsFilter.java:38)
        at coldfusion.filter.DatasourceFilter.invoke(DatasourceFilter.java:22)
        at coldfusion.CfmServlet.service(CfmServlet.java:175)
        at coldfusion.bootstrap.BootstrapServlet.service(BootstrapServlet.java:89)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:86)
        at coldfusion.monitor.event.MonitoringServletFilter.doFilter(MonitoringServletFilter.java:42)
        at coldfusion.bootstrap.BootstrapFilter.doFilter(BootstrapFilter.java:46)
        at jrun.servlet.FilterChain.doFilter(FilterChain.java:94)
        at jrun.servlet.FilterChain.service(FilterChain.java:101)
        at jrun.servlet.ServletInvoker.invoke(ServletInvoker.java:106)
        at jrun.servlet.JRunInvokerChain.invokeNext(JRunInvokerChain.java:42)
        at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:286)
        at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)
        at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)
        at jrunx.scheduler.ThreadPool$DownstreamMetrics.invokeRunnable(ThreadPool.java:320)
        at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)
        at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
        at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

Any suggestions on where to look for clues would be greatly appreciated.

    This topic has been closed for replies.

    1 reply

    C
    carl type3
    Legend
    February 15, 2011

    Hi,

    First - complements on the forum entry for the detail.

    I think this is the issue - java.lang.OutOfMemoryError: GC overhead limit exceeded

    While you can change a setting in CF Java Virtual Machine to increase the threshold before a GC overhead limit is reached I think it is better to change some other parameters to stop the JVM getting in to a state of doing so many Full or Major Garbage Collections in a sort time interval. The GC’s are perhaps because of a heap or non heap minimum / maximum setting. Perhaps details of your JVM args in ColdFusion\runtime\bin\jvm.config will be good to know?

    While I could make a guess at the heap or non heap it might be good to enable some JVM logging to find out what is attributing to the frequent GC. Let me know if you want details on how to enable logging?

    I might add most of my experience with this is CF on Win and IIS however we have the Java VM in common so likely the methodology is the same.

    HTH, Carl.

    G
    GuitsBoyAuthor
    Inspiring
    February 15, 2011

    Thank you for the reply.

    Yes, I agree that the garbage collection limit seems to be the issue. I have a ton of these errors in the web1 server logs, however very few in the logs on the other four servers.

    My JVM args are:

    -server -Dsun.io.useCanonCaches=false -XX:MaxPermSize=192m -XX:+UseParallelGC -Dcoldfusion.rootDir={application.home}/../ -Dcoldfusion.libPath={application.home}/../lib

    This is the same as the other four web nodes.  I currently have Min heap size set to NULL and Max heap size set to 512MB.  I have tried upping this to min 512 and max 2048 with no luck.  Again, the sefault min NULL and max 512 is the same as the other servers.

    As far as enabling logging, a quich google search leads me to believe it's as simple as adding a few arguments to the jvm.config such as:

    -XX:+PrintGCDetails -XX:+PrintGCTimeStamps  -XX:+PrintHeapAtGC -verbose:gc -Xloggc:PROD_GC.log

    If this is not correct, I would very much appreciate a little help on enabling logging and what to look for.

    Thanks you again for the help,

    -Tony

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices