ColdFusion 9 sites going down regularly once per day

Report · Sep 14, 2016

Hello,

I'm new to the forum, so please bare with me if I have not provided the proper details or in the desired format.

Problem: Daily, usually in the evening or early morning hours, USA time, our Cold Fusion sites are found down (4 sites, plus the default admin, cf instances) all connected through IIS and the Cold Fusion sites are configured to run as a Windows Service.

Resolution: Start the sites via jrun.exe or start them via Windows Services (services.msc) and it works fine, no further issues until the next problem occurs again later.

System:
Windows 2008 Enterprise 64bit
VMware client running on VMHost - HOST info: VMware ESXi 5.1.0 64bit, 1 CPU, 2 cores assigned. 6GB of RAM. VMware host reports less than 50% CPU usage and memory across multiple VM Clients (Guests), no spikes of note.
Cold Fusion 9 64bit

Cold Fusion:
Server Product ColdFusion
Version 9,0,1,274733
Edition Enterprise
Operating System Windows Server 2008
OS Version 6.1
Adobe Driver Version 4.0 (Build 0005)

Server has one a week backups, once monthly bounces, patches applied one a month on the server. We have a production service, much higher usage, no problems.

Environment: Dev/test server, small team of users

Users starting the service/sites via JRUN, have admin rights on the server.

Storage is configured as SAN, as the F:\ drive on the server (17GB of 300GB free). No known processes are running at this time. Windows Security, Application, System, and Setup logs show only one error:

"The description for Event ID 0 from source ColdFusion 9 ODBC Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

ColdFusion 9 ODBC Server@LOCALHOST,ErrorCode=2300,ErrorMessage=Network problem, connection unexpectedly closed by peer"

Anyone have any thoughts as to what I can try or how can I track to see what is happening when this occurs (other than physically watching for it to happen) or if there is a known issue I haven't found?

Thanks so much,

DB

Report · Sep 14, 2016

Hi Dean,

Perhaps there will be some useful information in some CF logs.

[CF]\runtime\logs server or [Jrun]\logs multiserver coldfusion-event.log & coldfusion-out.log

check for -

•java.lang.OutOfMemoryError: GC overhead limit exceeded

•java.lang.OutOfMemoryError: Java heap space

•java.lang.OutOfMemoryError: PermGen

•java.lang.StackOverflowError:

•java.lang.OutOfMemoryError: Out of swap space

CF9.0.1 so perhaps Java 6 in use? CF9 some update levels (9.0.1 being one as best I recall) officially

supports Java 7. It might not be much of an upgrade to change from EOL Java 6 to EOL Java 7 but could help.

As I recall CF9 does not support current Java 8.

HTH, Carl.

Report · Sep 15, 2016

Hi Carl, thank you for taking the time. Will engage developers to consider the same suggestion to upgrade. Production running on like configuration/hardware has zero issues.

The only recurring error I see in the event log (cfusion and all configured sites) is:

error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@7b0acf26]) JRunPRoxyServer.invokeRunnable:

java.lang.IllegalStateException

at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

...this is a recurring issue in the logs. I regret that I did overlook it as a possible cause, but is it the error I should be focusing on? This is the only error that recurs in the logs

and of course in the outlog:

Information [scheduler-0] - Run Client Storage Purge recurs about every 1 to 1.5 hours

In the production system, the output is pretty much the same on the outlog, but the event log does not have this above issue.

And again, I was focusing on general cold fusion since all sites cashed, but when I started looking into the extra sites, I found this:

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

at coldfusion.compiler.IntegerStack.<init>(JJTcfml40State.java:203)

at coldfusion.compiler.JJTcfml40State.<init>(JJTcfml40State.java:34)

at coldfusion.compiler.cfml40.<init>(cfml40.java:9)

at coldfusion.compiler.EvaluateEngine.parse(EvaluateEngine.java:35)

at coldfusion.compiler.ExprClassLoader$StatementCache.fetch(ExprClassLoader.java:73)

at coldfusion.util.LruCache.get(LruCache.java:180)

at coldfusion.compiler.ExprClassLoader$ExprCache.fetchSerial(ExprClassLoader.java:362)

at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

at coldfusion.util.SoftCache.get(SoftCache.java:81)

at coldfusion.compiler.ExprClassLoader.compileStatement(ExprClassLoader.java:222)

at coldfusion.compiler.ExprClassLoader.evaluate(ExprClassLoader.java:143)

at coldfusion.runtime.CFPage.Evaluate(CFPage.java:1759)

at cfact_scrolling2ecfm421729402.runPage(F:\inetpub\wwwroot\_Common\act_scrolling.cfm:8

I'm checking the GC settings now and will adjust and test.

Anything else you might suggest? And thanks again.

Dean

Report · Sep 15, 2016

Hi Dean,

Following up this:

JRunPRoxyServer.invokeRunnable java.lang.IllegalStateException

I searched and found this old CF9 thread where that turned out to be something in CFC code.

Here is the post. See details refering to - changed the following in my application.cfc

https://forums.adobe.com/thread/776671

I see you have this:

java.lang.OutOfMemoryError: GC overhead limit exceeded

JVM configuration can sometimes help with that.

Regards, Carl.

Report · Sep 15, 2016

Hi Carl,

Thanks to you putting me on possibly the right path, I have changed the Xms/Xmx settings to 256/384M. On this server, there are 6 sites running and only 6GB of RAM. At this value, with other processes, we gobble up 80% of physical memory. If I bump it up to 512M, it hits 99%. I think 384M, if they want to run all 6 sites, will be the limit.

I will also add this other value you suggested and report back (hopefully tomorrow) with an answer.

Thank you again sir.

Dean

Report · Sep 15, 2016

Hi Dean,

Well I am not sure what I have got to say will be helpful. I note CF9 64 bit so the Java JRE in use will be 64 bit as well. You can appreciate 384Mb on 32 bit is fair to say moderate on 64 bit I tend to think it is not near enough. As you point out with 6Gb total there is little to no wiggle room. Maybe some JVM tweaks can be applied to make the small heap evacuate frequently.

Regards, Carl.

Report · Sep 15, 2016

Hi Dean,

Perhaps enable some Java logging to see what the heap and non heap is doing then knowing that some kind if JVM tune can be figured out.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

Creates a log file in ColdFusion\runtime\bin\cfjvmgc.log Jrun4\bin\ in case multiserver

HTH again, Carl.

Report · Sep 19, 2016

Thanks Carl - I will try that too.

Dean