A

Anonymous

Question

"Java heap space" error and coldfusion crashes

Forum|Forum|15 years ago
March 16, 2010
5 replies
33181 views

I have a coldfusion scheduled task that takes about two hours to run (about 4000 lines of code). It does a number of cfquery, cffile, cfexecute with cffunction, cfloop, and cfif. If works perfectly in the fifteen minute version that works with a subset of the data, but it frequently runs coldfusion out of java heap in the full record set run. This then sometimes causes our public webserver to go down or malfunction until the coldfusion process is restarted.

When I posted about getting this to work initially last year, the solution was to put output="no" in every cffunction as apparently the whitespace generated by the code in all the cfloops was eventually enough to crash the server after a couple hours.

Recently, it has started to crash again as the record count has grown over the past year. I put everything in a number of cfsilent tags, and it ran for one night, but now it is crashing again.

I'm not sure what to do next. The cfexecutes log to a file not to a variable. The variables are in cffunction with var keyword with output="no" on the cffunction with coldfusion debugging checkboxes unchecked in the admin so in theory the fifteenth step/cffunction in the process should have the same memory available after garbage collection if needed as the first. But obviously it doesn't as each step can run by itself but not sequentially unless limited in record count.

Any ideas? Any known coldfusion memory leaks in 8,0,0,176276 Standard? Any way to prevent coldfusion server from malfunctioning if one page runs it out of memory?

Getting started

This topic has been closed for replies.

XeeMe

Participating Frequently

We have the same issue and it appears as we look around in the web that the garbage collection isn't designed very well. amybe ask to fix it in CF11 http://coldfusion.uservoice.com

C

carl type3

Brainiac

Wow Rob waking up a March 2010 thread. CF is Java and java garbage collects. There are many ways you can adjust CF (java) memory and GC routines. Best to do some kind of logging or analysis to know what is going on, then armed with knowing what is happening make an adjustment to memory, GC or both and monitor some more.

Regards, Carl.

I

ifsteve

Inspiring

I know this is an old threat, but its the most detailed I've seen and touches on something I am looking for.

I've been experiencing an issue where the mail spool crashes and stops delivering cfmail until the mail service is restarted or CF is restarted. Upon investigation, in my most recent crash I noticed that at the same time the spool stopped, I had a GC limit issue. I just doubled my heap size from 512 to 1024. I have 16gb of ram on the server and task manger's performance tells me I'm using no more than 6.5gb.

How would someone go into more detail to find out what would cause the GC issue?

A

Anonymous

It started crashing again, but I believe I solved it partially. Here's how I worked the problem and what I did in the end.

I had my company's credit card in hand ready to purchase FusionReactor at yalls suggestion, but after a couple discussions with a couple people at FusionReactor it was revealed that it does not do heap analysis (beyond current state summary numbers) so it would only reveal which thread was using up the memory. It might hint at memory usage detail when combined with FusionDebug when stepping through lines of code and looking at changes in summary numbers, but luckily there was a better path than buying two $299 products to hint at the answer. I might add that they are quite nice products that I'm consdering buying anyway, but they are not a slam dunk solution to needing to determine coldfusion memory usage detail when everything is pointing to ColdFusion internals holding the memory.

What I did was I added

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:/memdump

to the Coldfusion JVM arguments.

Then when it crashed I used Eclipse with MAT plugin to browse the heap dump. As expected, the heap dump was not interpreted (i.e. you couldn't tie memory very easily to a coldfusion variable by name). It showed the memory was being used up by char[] variable types. The largest one was one that begun with whitespace for a particular thread so I assumed it was page output. Since I already had dozens of <cfsilent>'s and almost all code inside of <cffunction> with output="no", this was troubling.

What I ended up trying was adding:

<cfsetting enablecfoutputonly="true"> at the beginning of pages

and

<cfsetting enablecfoutputonly="false"> at the end of pages (it's a peculiar tag but this is apparently the best practices for this setting)

and then adding <cfoutput> around where I was worried it inappropriately might not generate special-purpose output (e.g. body of cfmail, etc.)

Apparently this helped Coldfusion to not run itself out of memory despite already being inside both a cfsilent and inside a cffunction with output="no".

There was another mysterious large char[], but I was unable to determine what that was. Wish there was a heap browsing tool for ColdFusion that fully inteprets everything!

A

Anonymous

I'm not consuming memory with left over variables (at least ones I've explicitly created). As I already said, all the relevant variables are in a cffunction with the var keyword. This is occasionally verified by scope cfdumps outside of the cffunction scope. I've been staring at this code trying to optimize it for memory for a year so please give me a little creditibility.

Splitting up the sequential process into multiple requests is a complication with consequences which should be unnecessary and is not desired. I appreciate your efforts to find me a workaround though, and in fact I broke the sequential process into three scheduled tasks which is stopping the server crashes for the moment. I believe this to be only temporary fix though.

I agree Coldfusion is not the best tool for the job but it should be adequate. Eventually I'll rewrite it in Java. As programmers, we often don't have control over what language we write in. It has to be on an outward facing server as there are webservices part of the application, but I'm considering an outward facing server other than our primary webserver.

Wish I had Enterprise, but I don't have it. Dropping $7K instead of $1K for the same product is a hard sell at my company.

I don't have FusionReactor either. It looks like a nice program, but I can't justify spending $299 on software that I would use for an hour and might not even help. Not sure I want to install a trial version on our production server either.

I think this is what I'm looking for:

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=D:\heapdumps

(combined with jhat from the JDK)

Anyone ever try this with ColdFusion and find it detailed and understandable enough to determine what is running ColdFusion out of memory? I'm dreading coming in on off hours to try it out and it telling me that there are 300,000,000 instances of "java.util.string" with little hint to what is causing them..

ilssac

Inspiring

Point One: All systems have limits. Sometimes you just have to say that this is all that we can do with the tools we have to work with. To quote Mr. Scotty, "I can not change the laws of physics Captain!"/

Point Two: ColdFusion is optimized as a web application server. As such it focuses on the request as an atomic unit. It does not naturally try to do overhead, housekeeping tasks, such as garbage collection, during a request.

You can either try to break up your tasks into multiple requests to work within this framework, or you can try other garbage collection and JVM memory management settings. But realize with the latter option you are choosing to optimize your CF to handle this scheduled task, which could have performance consequences to the day to day web severing tasks the server is also expected to perform.

T

TLC-IT

Inspiring

Is there any other tool that you could use to accomplish this task? If the ultimate objective of the task is not "to generate HTML output," it might be argued that you are simply using the wrong tool for the job -- even tough ColdFusion is, after a manner of speaking, doing it.

A very definite possibility here is that you are somehow building-up data within (e.g. global...) variables or hashes (structs...) that never get cleaned away. They never get garbage-collected because they never become "un-referenced."

One strategy that might help alleviate the situation is to put all of your variable references into global structures, e.g. named GLOBAL and/or LOCAL. Then, at the top of each processing loop, you insert a statement like: <cfset GLOBAL = StructNew()> . Now, explicitly, the old structure (and everything in it) is thrown-away and a new empty structure is allocated. This happens each time through the loop. If you need to retain the values of certain variables, add explicit code at the top of the loop that sets-aside these variables (in another, temporary, struct), resets the main global struct, puts the variables back, and then resets the temporary one before proceeding.

One of the characteristics of ColdFusion, particularly when you are using CFML tags vs. <cfscript>s, is that it is very easy for everything to wind up in a global scope. Lots of things can hang-around in memory much longer than they need to. All of which might not matter much if you are serving web pages (and recycling the worker-threads every so often), but it becomes a crippling concern when a thread runs for a very long time, such as multiple hours.

A

Adam Cameron.

Inspiring

One of the characteristics of ColdFusion, particularly when you are using CFML tags vs. <cfscript>s, is that it is very easy for everything to wind up in a global scope.

Interesting. Do tell..?

--

Adam

ilssac

Inspiring

I'm not sure what the reference between the tag and script form or CFML means. But I presume he is refereeing to what I alluded to. That if you do not take steps to cause different behavior, a great deal of the data in a request will end up in the variables scope and this scope is not going to be dereferenced for garbage collection until the end of the request. Thus requests that run for a very long time can become problematic.

A

Anonymous

I might add that I've done everything justifyable already to upgrade the memory. The box has more memory than the OS can use, and the OS has more memory that coldfusion will use. Coldfusion has the highest JVM heap settings possible while being able to start up. (1450 MB) . Beyond dropping $30k on a 64 bit server, 64 bit OS, and coldfusion enterprise, there's nothing I know of to do with hardware or configuration.

ilssac

Inspiring

IF you can't upgrade to enterprise you are definitely limited in options.

With Enterprise, whether 64bit or not, you could run multiple instances of ColdFusion. So you could have one instance with all of it's JVM memory dedicated to this resource consuming task, and another instance for your public web site. This would at least allow application isolation so that the problems and failures of this task does not affect the other applications.

But even with that, it is not going to solve the underlining problem that you are using up all the memory. There is always limits in programming. We have come a long way from the days where years were stored as two digits instead of four digits to save memory so that applications would not run out of it. But it is still possible to create tasks that just plain use more memory then is available.

When this happens you have to look at your application and see what you can do to make it more frugal with memory. The first thought I had is does all this process have to be done in one request? The main problem is that the JVM really can't do much with garbage collection in the middle of a request.

Once we had a task to create several hundred PDF reports on a nightly basis. When we tried to do all of this in a single request it would quickly use up all the memory. What we did was to break up the task into a series of process that would process a handful of reports and then using a simple META refresh, generate a new request to process the next batch. This allowed all the variables used in the previous refresh to go to garbage collection and eventually get cleared out while new requests where processing other reports. We could also throttle the processing with the time between META refreshes to allow the server some breathing room between requests.

This made the overall process run a while longer, but the server stopped failing in the middle of it. Seemed a good compromise to us.

ilssac

Inspiring

P.S. I don't know if a META refresh would work with a scheduled task. I don't think the built in browser that ColdFusion uses to make scheduled task http requests would understand or process the meta refresh.

But the same thing could be done with a master page that then makes a throttled series of <cfhttp....> http requests to a sub page that process a single batch at a time.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded