Skip to main content
D
deanb69509320
Participant
September 14, 2016
Question

ColdFusion 9 sites going down regularly once per day

  • September 14, 2016
  • 1 reply
  • 1096 views

Hello,

I'm new to the forum, so please bare with me if I have not provided the proper details or in the desired format.

Problem:  Daily, usually in the evening or early morning hours, USA time, our Cold Fusion sites are found down (4 sites, plus the default admin, cf instances) all connected through IIS and the Cold Fusion sites are configured to run as a Windows Service. 

Resolution:  Start the sites via jrun.exe or start them via Windows Services (services.msc) and it works fine, no further issues until the next problem occurs again later.

System: 
Windows 2008 Enterprise 64bit
VMware client running on VMHost - HOST info:  VMware ESXi 5.1.0 64bit, 1 CPU, 2 cores assigned.  6GB of RAM.  VMware host reports less than 50% CPU usage and memory across multiple VM Clients (Guests), no spikes of note.
Cold Fusion 9 64bit

Cold Fusion: 
Server Product  ColdFusion 
Version  9,0,1,274733   
Edition  Enterprise   
Operating System  Windows Server 2008   
OS Version  6.1   
Adobe Driver Version  4.0 (Build 0005)  

Server has one a week backups, once monthly bounces, patches applied one a month on the server.  We have a production service, much higher usage, no problems.

Environment:  Dev/test server, small team of users

Users starting the service/sites via JRUN, have admin rights on the server.

Storage is configured as SAN, as the F:\ drive on the server (17GB of 300GB free).  No known processes are running at this time.  Windows Security, Application, System, and Setup logs show only one error:

"The description for Event ID 0 from source ColdFusion 9 ODBC Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

ColdFusion 9 ODBC Server@LOCALHOST,ErrorCode=2300,ErrorMessage=Network problem, connection unexpectedly closed by peer"

Anyone have any thoughts as to what I can try or how can I track to see what is happening when this occurs (other than physically watching for it to happen) or if there is a known issue I haven't found?

Thanks so much,

DB

This topic has been closed for replies.

1 reply

C
carl type3
Legend
September 14, 2016

Hi Dean,

Perhaps there will be some useful information in some CF logs.

[CF]\runtime\logs server or [Jrun]\logs multiserver coldfusion-event.log & coldfusion-out.log

check for -

•java.lang.OutOfMemoryError: GC overhead limit exceeded

•java.lang.OutOfMemoryError: Java heap space

•java.lang.OutOfMemoryError: PermGen

•java.lang.StackOverflowError:

•java.lang.OutOfMemoryError: Out of swap space

CF9.0.1 so perhaps Java 6 in use? CF9 some update levels (9.0.1 being one as best I recall) officially

supports Java 7. It might not be much of an upgrade to change from EOL Java 6 to EOL Java 7 but could help.

As I recall CF9 does not support current Java 8.

HTH, Carl.

D
deanb69509320Author
Participant
September 15, 2016

Hi Carl, thank you for taking the time.  Will engage developers to consider the same suggestion to upgrade.  Production running on like configuration/hardware has zero issues.

The only recurring error I see in the event log (cfusion and all configured sites) is:

error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@7b0acf26]) JRunPRoxyServer.invokeRunnable:

java.lang.IllegalStateException

at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

...this is a recurring issue in the logs.  I regret that I did overlook it as a possible cause, but is it the error I should be focusing on?  This is the only error that recurs in the logs

and of course in the outlog:

Information [scheduler-0] - Run Client Storage Purge recurs about every 1 to 1.5 hours

In the production system, the output is pretty much the same on the outlog, but the event log does not have this above issue.

And again, I was focusing on general cold fusion since all sites cashed, but when I started looking into the extra sites, I found this:

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

at coldfusion.compiler.IntegerStack.<init>(JJTcfml40State.java:203)

at coldfusion.compiler.JJTcfml40State.<init>(JJTcfml40State.java:34)

at coldfusion.compiler.cfml40.<init>(cfml40.java:9)

at coldfusion.compiler.EvaluateEngine.parse(EvaluateEngine.java:35)

at coldfusion.compiler.ExprClassLoader$StatementCache.fetch(ExprClassLoader.java:73)

at coldfusion.util.LruCache.get(LruCache.java:180)

at coldfusion.compiler.ExprClassLoader$ExprCache.fetchSerial(ExprClassLoader.java:362)

at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

at coldfusion.util.SoftCache.get(SoftCache.java:81)

at coldfusion.compiler.ExprClassLoader.compileStatement(ExprClassLoader.java:222)

at coldfusion.compiler.ExprClassLoader.evaluate(ExprClassLoader.java:143)

at coldfusion.runtime.CFPage.Evaluate(CFPage.java:1759)

at cfact_scrolling2ecfm421729402.runPage(F:\inetpub\wwwroot\_Common\act_scrolling.cfm:8

I'm checking the GC settings now and will adjust and test.

Anything else you might suggest?  And thanks again.

Dean

C
carl type3
Legend
September 16, 2016

Hi Carl,

Thanks to you putting me on possibly the right path, I have changed the Xms/Xmx settings to 256/384M.  On this server, there are 6 sites running and only 6GB of RAM.  At this value, with other processes, we gobble up 80% of physical memory.  If I bump it up to 512M, it hits 99%.  I think 384M, if they want to run all 6 sites, will be the limit.

I will also add this other value you suggested and report back (hopefully tomorrow) with an answer.

Thank you again sir.

Dean


Hi Dean,

Perhaps enable some Java logging to see what the heap and non heap is doing then knowing that some kind if JVM tune can be figured out.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

Creates a log file in ColdFusion\runtime\bin\cfjvmgc.log Jrun4\bin\ in case multiserver

HTH again, Carl.

Terms & ConditionsAccessibility statement
Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices