Java Heap OutOfMemory and GC

Forum|Forum|18 years ago
September 11, 2007
7 replies
2320 views

We have two applications on the same server - one uses an Oracle data source and the other a SQL Server source. When the Oracle app runs, it uses heap space but GC periodically runs and gives back memory. However when the SQL Server app runs, the heap holds on to more and more memory until it runs out of space.

These two apps run very similar code but simply point to different data sources. I have read many forum entries regarding OutOfMemory errors but this may be different because even if the code doesn't allow memory to be freed, the heap is eventually freed when CF times out and figures the app is gone (the standard 20 minute timer). That happens with the Oracle app but not the SQL Server app. Even when the "Session Timeout" variable is reached the heap memory is still held. The only way to recover that memory is by restarting CF.

This is a RedHat AS 4 Linux machine and it happens with both CFMX7 and CF8 in single server format. I've watched the memory charts with CF8 (very cool) and you can see the peaks and valleys for the Oracle run but the SQL Server app grows and grows.

I have turned on Heap logging and watched several Young GC iterations followed by a Full GC. That pattern continues while the Oracle app is running. However the SQL Server app starts like that but as time goes on it resorts to only Full GCs until it gets to the "GC time is exceeding GCTimeLimit of 98%" errors and "java.lang.OutOfMemoryError: Java heap space" errors.

So my questions are: given that the apps run differently on the same machine, could there be a difference between the Oracle and SQL Server data sources and why doesn't CF ever find the application is gone and give back some heap space?

This topic has been closed for replies.

Correct answer hoosier102

I found the answer and unfortunately I was wrong in my original assumptions. Turns out the code was different from one application to the other since one of them uses "cachedwithin" and the other doesn't.

The developer thought he had removed that code as a possible problem when he used zeroes as parameters (as in "cache for 0 days, hours, minutes, and seconds"). However, it still cached the same data. But when he removed the cachedwithin paramater completely, then nothing was cached and the heap stayed calm.

So we can see that we'll have to be much more careful about what gets cached and when and for how long. A listing of exactly what goes into each part of the Java heap would be nice (session variables, cfcache data , cachedwithin, etc)....

H

hoosier102Author

Participant

Your last thought - yes, I had tried "maintain connections" both on and off. It seemed like a possibility.

I tried some other GC variables that I had read about (like "UseParNewGC " and "UseConcMarkSweepGC"). But of course this didn't help at all - it just uses different ways to not collect what it wasn't collecting before.

I guess I'll concentrate on the machine running the DB, as well as its SQL Server config. I hope to try a SQL Server 2005 machine soon. It just seems like something someone would have run into before while using a SQL Server source.

H

hoosier102AuthorCorrect answer

Participant

I found the answer and unfortunately I was wrong in my original assumptions. Turns out the code was different from one application to the other since one of them uses "cachedwithin" and the other doesn't.

The developer thought he had removed that code as a possible problem when he used zeroes as parameters (as in "cache for 0 days, hours, minutes, and seconds"). However, it still cached the same data. But when he removed the cachedwithin paramater completely, then nothing was cached and the heap stayed calm.

So we can see that we'll have to be much more careful about what gets cached and when and for how long. A listing of exactly what goes into each part of the Java heap would be nice (session variables, cfcache data , cachedwithin, etc)....

G

Grizzly9279

Participating Frequently

One last thought - have you looked into the connection pooling settings for the SQL Server data source in question? In the CF Administrator, under the DSN configuration, is "Maintain Connections" enabled?

I believe it should be enabled by default...but you never know.

G

Grizzly9279

Participating Frequently

Yeah, the multi-server installation allows you to configure and deploy multiple ColdFusion applications on the same server, each having their own independent JVM.

I don't anticipate that fixing your problem, but you could at least isolate the memory hogging application from others on the server so it doesn't negatively impact other sites.

Anyways, best of luck with it!

H

hoosier102Author

Participant

Okay I'm holding off on different JVMs.

I did, however, load the latest SQL Server 2005 JDBC jar file. I created a new ('other') Data Source out of it and tried it again. I was encouraged at first because memory management seemed better but over the length of the test it again ran out of memory.

Thanks to the following sites for help on setting up SQL 2005 JDBC:
-> http://www.fusioncube.net/?p=59
-> http://kb.adobe.com/selfservice/viewContent.do?externalId=ded4216b&sliceId=1

Even though it didn't help, it does get me set up for when they upgrade the database to 2005.

I am still trying to find out from the DBA about TCP/IP versus named-pipes. In the meantime, I wonder if reinstalling CF in the Multiserver configuration might help since that employs its own JRUN server rather than an embedded one. Or are they virtually the same thing and I'd be wasting my time?

G

Grizzly9279

Participating Frequently

Yeah it's hard to say really. If memory is truly being "leaked", than the GC won't have any ability to reclaim it (as far as I know).

If what you say is true, and the only significant difference between the two applications is the database that they point to, than the fault must lie in the DB connection methods. (You're sure that the SQL Server application isn't using query-caching or anything along those lines correct?)

If you could run the exact same tests against the SQL Server application using different DB connection methods (or even different DBs as you suggest), than that might be a good way to begin isolating the problem.

Personally, I wouldn't start trying different JVM versions, unless all other avenues had been tried first.

Do you know if the SQL Server is question is accepting connections via TCP/IP, or named-pipes? That may be one place you can also look, as named-pipes are usually your best bet (I've battled issues related to this many years ago). Although this article is about SQL Server 7, it might be relevant to your problem: http://support.microsoft.com/kb/q236439/

Best of luck...these types of issues are tough to tackle, but once you get it straightened out you'll feel like you've earned your paycheck(s) :)

H

hoosier102Author

Participant

I agree, it does sound like a leak. But wouldn't Java reclaim the memory once the SQL Server link went away? Otherwise I would blame the code or even CF itself.

I am trying to set up another connection, this time to SQL Server 2005 (the other is SQL Server 2000). I saw a forum post that said "You must use SP3 for CF and SQL Server" but is that 2000 or 2005?

I thought about trying a different JVM (like IBM's) as I have seen suggested but since the Oracle version works with Sun JVM (1.6_01) then it seems that the SQL Server should.

G

Grizzly9279

Participating Frequently

It sounds like there's a memory leak there somewhere related to the SQL Server connection. Are you able to experiment with "other" methods of connecting to the SQL Server database? (e.g...different drivers, or different protocols altogether)

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded