Pages failing with Java Heap errors

[Guest]

We are running CF 8,0,0,176276 on a Windows 32 bit 2003 server, with IIS 6 and Java Runtime version 6 ( it's really version 1.06).
We started getting the java heap error messages two weeks ago. Now we are also getting the 'GC overhead limit exceeded' error, which is concerned with the Java garbage collection. I tried to change the JVM heap size from 1GB to 2GB, but CF wouldn't start up. I've since found out that Windows 32bit servers can't use 2GB for Java.
One java article I found said to turn off the Java garbage collection using -XX:-UseGCOverheadLimit ( http://forums.java.net/jive/message.jspa?messageID=228596). It seems like turning off the garbage collection would be the opposite of what I am trying to do. Are there any thoughts about this?
Are there any hotfixes for CF8? and where are they located?
Scott

[Charlie Arehart · Accepted solution]

Scott, the error is a symptom, not the cause of your problems. Turning off that option won't "solve" the problem, which is that you're running out of memory. More on that option in a moment.

Let's talk first about your observation of trying to change the max memory from 1 to 2 gig. Since you're on Windows (32 bit), as you say, you can't use more than 2g per process (it's not really a Java limit, but rather a Windows one).

But as you found, you can't set CF to 2g because the JVM won't start. Again, this is not a CF-specific issue. You can see discussions of the same problem with plain old Java apps, as at http://forums.java.net/jive/message.jspa?messageID=228596.

So, similar to what they say, the consensus seems to be that you can set it to somewhere between1.3 and 1.7 gb, before the JVM won't start. You have to try different values for yourself.

Now, that extra few hundred meg may be all you need to get past your problem, but really that's often just forestalling the inevitable. You need to find why CF is using so much memory. You say you're using CF8 but don't say if it's Enterprise or Standard. If the former, then you have the CF8 Server Monitor (accessible in the CF Admin, and discussed in a multipart series I wrote for the Adobe Dev Center, as discussed at http://carehart.org/blog/client/index.cfm/2008/7/30/45page_server_monitor_guide). That, as well as tools like FusionReactor and SeeFusion, can help you to better see how CF is using memory.

All three offer a graphical interface to show how memory use is trending. You may see if it grows steadily throughout the day, or grows in spikes. People tend to say "I have a memory leak", when in fact it may be just that they have either a few requests that use too much memory (so you'll see a stair stepping rise in memory use concurrent with their execution), or it may be that they're sticking stuff into shared scopes (session, application, or server) and those are living well beyond the live of each request.

The CF8 Server Monitor even has a memory tracking feature, but it has to be used carefully. If you enable it, you may find it brings your server to its knees.I'll do a blog entry soon on how to use it most effectively (something I learned since writing the articles last year).

One thing that the monitor can tell you, without any of the "start" buttons enabled at the top of the monitor, is how many sessions you have. That may be very enlightening. You may have thousands of them, and not realize it, because of bots, spiders, and other crap requests. See the "active sessions" page of the Server Monitor, and its graph icon in particular (discussed in more detail in one of my articles).

Sadly, neither FusionReactor nor SeeFusion can tell you how many sessions are currently running, so if you're on CF8 Standard (or if others reading this are on 6 or 7, where they can use FR or SF), you can't easily see how many sessions there are. (There are undocumented, unsupported methods, such as discussed at http://rewindlife.com/2003/09/08/undocumented-application-scope-functions/. Just be aware some techniques work differently depending on whether J2EE sessions are enabled in th CF admin.)

Finally, about your observation of the usegcoverheadlimit option, and the recommendation to turn it off: the article you point to doesn't say doing so would disable GC. I'm curious how you may have concluded that.But digging into things more, I see that disabling it (using the - before it) will turn off a feature (introduced in 6) which limits the proportion of the VM's time that is spent in GC before an OutOfMemory error is thrown. That's the error you're seeing, that the JVM has detected too much time is being spent in doing GC.

So turning off the option doesn't "turn off the Java garbage collection". It just changes whether an OOM error will be thrown even before being really out of memory, simply because too much time is being spent in GC.

And even then, how this option works (if enabled, which it is by default) changes depending on whether you're using a parrallel or concurrent collector (as discussed in two places in http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html, if you search for that switch).

Hope some of that helps.

[Guest]

Charlie,
Thank you very much for explaining something that was very difficult for me to understand. So I did jump to the wrong conclusion about the java garbage collect switch. If a user starts a session that uses quite a bit of java memory, and then they end their session, there will be quite a bit of garbage collection happening. So even though the amount of available java memory is increasing, a java memory error will be thrown because the garbage collection process is too active? Am I close on that? I would never have understood that from the java article.
I do have CF8 Enterprise edition so I'll try the Server Monitor. Thank you for the very practicle manual on the use of the server monitor.
And thank you for being the chair of the CF meetup boards http://coldfusion.meetup.com/17/boards/ They present some very good information about different things that people are doing with ColdFusion.
Scott

[Charlie Arehart]

Glad to hear it may have been helpful, Scott. As for the GC error, well, I wouldn't say it was just because the GC process was "too active". It's always very active. But that particular error seems tied to that particular feature (new in 6) where if it determines that the JVM is spending "too much time" in GC that it will mark it as an out of memory problem. I've not heard of it happening yet with any other clients of mine. Yours was my first observation of it.

Now, you propose that there may be a good bit of GC happening when sessions end, and that may be possible, but it should be pretty random that that happens, so I wouldn't call it a smoking gun. Really, I'd bet more junk is released to be GC'ed simply at the end of every request.

There are ways to ask the JVM to log GC details to tell you when it's happening, and how often. It tends to be quite often with modern JVMs, but in different parts of the memory space, which is usually a good thing.

But maybe rather than focus on how the JVM is operating, you might instead want to look at the bigger picture. Are you using lots of memory? It may or may not be possible to tell from the task manager, btw. The Server Monitor will be your best bet. It has a page showing memory usage as a chart which updates live over time. And as I explain in the articles on the Server Monitor, you DO NOT need to enable "start memory tracking" to see that chart.

Now, if in fact you are using high amounts of memory, your mention of sessions, and the fact that you put a lot of info into the session when a user starts, is a real clue.

I'm curious: do you do that whether the user logs in or not? If you do so even if they do not log in, then this could very well be the cause of a memory usage problem.

Many don't think of this, but whenever a search engine spider, or other bots (or load test tool, or web service client), visits your site, they generally get a new session on *each page visit*, since they don't tend to present cookies, particularly the CFID/CFTOKEN/JSessionID that CF would have given them on their previous visit.

So on each visit to every page CF creates a new session for them (if you have sessionmanagement enabled in your application.cfm/cfc), and also a new client repository entry (if you have clientmanagement enabled).

It's a severe cause of problems for many people, in creating huge numbers of of sessions. Depending on what they do put in the sessions, and how long they set them to last, it could gobble up memory and hold it for a very long time. That's not a "memory leak", in the sense of bad code in the CF engine, but it can look like one.

It can also hurt them in the large number of client variable repository entries created--and it doesn't matter if one says, "well we don't use client vars". Doesn't matter. If you have clientmanagement enabled, CF will create an entry (unless you checked an option in the Admin client variable repository details, called "disable global client variable updates"), and whether you've set the registry or a database to be your repository (in the admin as default, or in code using the clientstorage attribute0, it could have another severe impact. I mention that just because the problems tend to go hand in hand, if you go exploring.

Thanks for your other kind words, too.