Welcome Dialog

Welcome to the Community!

We have a brand new look! Take a tour with us and explore the latest updates on Adobe Support Community.


ColdFusion 9 sites going down regularly once per day

New Here ,
Sep 14, 2016 Sep 14, 2016

Copy link to clipboard

Copied

Hello,

I'm new to the forum, so please bare with me if I have not provided the proper details or in the desired format.

Problem:  Daily, usually in the evening or early morning hours, USA time, our Cold Fusion sites are found down (4 sites, plus the default admin, cf instances) all connected through IIS and the Cold Fusion sites are configured to run as a Windows Service. 

Resolution:  Start the sites via jrun.exe or start them via Windows Services (services.msc) and it works fine, no further issues until the next problem occurs again later.

System: 
Windows 2008 Enterprise 64bit
VMware client running on VMHost - HOST info:  VMware ESXi 5.1.0 64bit, 1 CPU, 2 cores assigned.  6GB of RAM.  VMware host reports less than 50% CPU usage and memory across multiple VM Clients (Guests), no spikes of note.
Cold Fusion 9 64bit

Cold Fusion: 
Server Product  ColdFusion 
Version  9,0,1,274733   
Edition  Enterprise   
Operating System  Windows Server 2008   
OS Version  6.1   
Adobe Driver Version  4.0 (Build 0005)  

Server has one a week backups, once monthly bounces, patches applied one a month on the server.  We have a production service, much higher usage, no problems.

Environment:  Dev/test server, small team of users

Users starting the service/sites via JRUN, have admin rights on the server.

Storage is configured as SAN, as the F:\ drive on the server (17GB of 300GB free).  No known processes are running at this time.  Windows Security, Application, System, and Setup logs show only one error:

"The description for Event ID 0 from source ColdFusion 9 ODBC Server cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

ColdFusion 9 ODBC Server@LOCALHOST,ErrorCode=2300,ErrorMessage=Network problem, connection unexpectedly closed by peer"

Anyone have any thoughts as to what I can try or how can I track to see what is happening when this occurs (other than physically watching for it to happen) or if there is a known issue I haven't found?

Thanks so much,

DB

Views

713

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Sep 14, 2016 Sep 14, 2016

Copy link to clipboard

Copied

Hi Dean,

Perhaps there will be some useful information in some CF logs.

[CF]\runtime\logs server or [Jrun]\logs multiserver coldfusion-event.log & coldfusion-out.log

check for -

•java.lang.OutOfMemoryError: GC overhead limit exceeded

•java.lang.OutOfMemoryError: Java heap space

•java.lang.OutOfMemoryError: PermGen

•java.lang.StackOverflowError:

•java.lang.OutOfMemoryError: Out of swap space

CF9.0.1 so perhaps Java 6 in use? CF9 some update levels (9.0.1 being one as best I recall) officially

supports Java 7. It might not be much of an upgrade to change from EOL Java 6 to EOL Java 7 but could help.

As I recall CF9 does not support current Java 8.

HTH, Carl.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 15, 2016 Sep 15, 2016

Copy link to clipboard

Copied

Hi Carl, thank you for taking the time.  Will engage developers to consider the same suggestion to upgrade.  Production running on like configuration/hardware has zero issues.

The only recurring error I see in the event log (cfusion and all configured sites) is:

error (JRun Service: ProxyService [jrun.servlet.jrpp.JRunProxyService@7b0acf26]) JRunPRoxyServer.invokeRunnable:

java.lang.IllegalStateException

at jrun.servlet.JRunResponse.getWriter(JRunResponse.java:205)

at jrun.servlet.JRunResponse.sendError(JRunResponse.java:597)

at jrun.servlet.JRunRequestDispatcher.invoke(JRunRequestDispatcher.java:328)

at jrun.servlet.ServletEngineService.dispatch(ServletEngineService.java:543)

at jrun.servlet.jrpp.JRunProxyService.invokeRunnable(JRunProxyService.java:203)

at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:428)

at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)

...this is a recurring issue in the logs.  I regret that I did overlook it as a possible cause, but is it the error I should be focusing on?  This is the only error that recurs in the logs

and of course in the outlog:

Information [scheduler-0] - Run Client Storage Purge recurs about every 1 to 1.5 hours

In the production system, the output is pretty much the same on the outlog, but the event log does not have this above issue.

And again, I was focusing on general cold fusion since all sites cashed, but when I started looking into the extra sites, I found this:

javax.servlet.ServletException: ROOT CAUSE:

java.lang.OutOfMemoryError: GC overhead limit exceeded

at coldfusion.compiler.IntegerStack.<init>(JJTcfml40State.java:203)

at coldfusion.compiler.JJTcfml40State.<init>(JJTcfml40State.java:34)

at coldfusion.compiler.cfml40.<init>(cfml40.java:9)

at coldfusion.compiler.EvaluateEngine.parse(EvaluateEngine.java:35)

at coldfusion.compiler.ExprClassLoader$StatementCache.fetch(ExprClassLoader.java:73)

at coldfusion.util.LruCache.get(LruCache.java:180)

at coldfusion.compiler.ExprClassLoader$ExprCache.fetchSerial(ExprClassLoader.java:362)

at coldfusion.util.AbstractCache.fetch(AbstractCache.java:58)

at coldfusion.util.SoftCache.get_statsOff(SoftCache.java:133)

at coldfusion.util.SoftCache.get(SoftCache.java:81)

at coldfusion.compiler.ExprClassLoader.compileStatement(ExprClassLoader.java:222)

at coldfusion.compiler.ExprClassLoader.evaluate(ExprClassLoader.java:143)

at coldfusion.runtime.CFPage.Evaluate(CFPage.java:1759)

at cfact_scrolling2ecfm421729402.runPage(F:\inetpub\wwwroot\_Common\act_scrolling.cfm:8

I'm checking the GC settings now and will adjust and test.

Anything else you might suggest?  And thanks again.

Dean

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Sep 15, 2016 Sep 15, 2016

Copy link to clipboard

Copied

Hi Dean,

Following up this:

JRunPRoxyServer.invokeRunnable java.lang.IllegalStateException

I searched and found this old CF9 thread where that turned out to be something in CFC code.

Here is the post. See details refering to  - changed the following in my application.cfc

<cfset This.Clientmanagement="false">

https://forums.adobe.com/thread/776671

I see you have this:

java.lang.OutOfMemoryError: GC overhead limit exceeded

JVM configuration can sometimes help with that.

Regards, Carl.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 15, 2016 Sep 15, 2016

Copy link to clipboard

Copied

Hi Carl,

Thanks to you putting me on possibly the right path, I have changed the Xms/Xmx settings to 256/384M.  On this server, there are 6 sites running and only 6GB of RAM.  At this value, with other processes, we gobble up 80% of physical memory.  If I bump it up to 512M, it hits 99%.  I think 384M, if they want to run all 6 sites, will be the limit.

I will also add this other value you suggested and report back (hopefully tomorrow) with an answer.

Thank you again sir.

Dean

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Sep 15, 2016 Sep 15, 2016

Copy link to clipboard

Copied

Hi Dean,

Well I am not sure what I have got to say will be helpful. I note CF9 64 bit so the Java JRE in use will be 64 bit as well. You can appreciate 384Mb on 32 bit is fair to say moderate on 64 bit I tend to think it is not near enough. As you point out with 6Gb total there is little to no wiggle room. Maybe some JVM tweaks can be applied to make the small heap evacuate frequently.

Regards, Carl.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guide ,
Sep 15, 2016 Sep 15, 2016

Copy link to clipboard

Copied

Hi Dean,

Perhaps enable some Java logging to see what the heap and non heap is doing then knowing that some kind if JVM tune can be figured out.

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintHeapAtGC -verbose:gc -Xloggc:cfjvmGC.log

Creates a log file in ColdFusion\runtime\bin\cfjvmgc.log Jrun4\bin\ in case multiserver

HTH again, Carl.

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 19, 2016 Sep 19, 2016

Copy link to clipboard

Copied

LATEST

Thanks Carl - I will try that too.

Dean

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines