Garbage Collection Not Firing

[Guest]

We are having a bit of time with running out of memory. We have an application used by several hundred people at a time. It's fairly complicated OLTP business logic. We moved from CF 5 to CF 8 last fall. We are currently running 8.0 on a VM, Windows 2003.

The JVM settings were left alone, with the difference that I set maximum memory to 1024, and starting memory to 512.

What seems to happen is the used memory slowly climbs, over the space of an hour or two. On occasion, it will continue to climb until we run out of memory, and the server locks up - we either have to restart coldfusion, or reboot the server.

If I click 'run GC' on the server monitor, memory usage immediately drops down to less than 200 meg. Then it slowly builds up again. I have created a scheduled script that runs every 2 minutes, that checks used memory - if it hits 700 meg, it runs garbage collection. We'll see if this solves the problem long term, but it's still a bandaid, - if the memory is ready to be garbage collect, why is it not going?

Any suggestions would be muich appreciated. Seems different people have this problem for different reasons. I must say, CF5 was extremely stable. CF8 has been a bit more painful.

[Guest]

An update - my script to run GC is working - but I'm still confused as to why it should be necessary.Ran once at 8 this morning, and at 8:45, it won't be long until it has to do it again.

Any thoughts, or additional information required, would be greatly appreciated.

[Guest]

You don't want GC to happen too often.  When GC happens everything stops until GC is done.

But, you've got to have it happen enough to not run out of space.

Consider using these two flags in your java.args:

-XX:+UseConcMarkSweepGC -XX:+UseParNewGC

Do not use UseConcMarkSweepGC on single processor boxes.

You could also look at adding the switch to force GC every so often.  Though, I think I read somewhere that was depricated in Java 1.6.

Finally, consider upgrading to the latest and greatest version of 1.6.  Lots of fixes after 1.6.0_10.

[Guest]

Thanks for the reply. The problem has evolved - either I did not have the original problem correct, or having my script firing and logging GC has just provided more info. GC only takes a second or so, so it's not a big problem - slow downs are not noticeable.

We are getting sudden steep memory climbs - it gets over the threshold, GC fires, but it reclaims hardly any memory. Memory continues to rise until the server is unresponsive, and we have to reboot ColdFusion. I'm not sure how to try and isolate what is happening, there's no real good information in the logs, and I don't seem to get much out of the server monitor, either. We are not using CFCs, except in our online text editor - rest is all old CFM from a CF5 installation.

Any other suggestions would be helpful. I'll look into updating the JVM.

[Guest]

Rick,

I forgot to put the obligatory "Get Fusion Reactor" in my last post.  It can be invaluable in tracking down stuff like this.

It may just come down to your app needing a lot of room to breath.  Options then are pretty much limited to going to 64 bit and opening up the JVM to a LOT of memory.  Our standard JVM heap setting now is 4096/4096.

[Guest]

Finally found it, I think. I looked in my Apache logs and found an error message, 'winnt_accept: Asynchronous AcceptEx failed', at the same time as the last two failures. I googled this, and found this post in a blog: http://www.brooks-bilson.com/blogs/rob/index.cfm/2008/1/4/Intermittent-Apache-Problems-and-winntacce.... This looked interesting, and this weekend I made the change in the apache conf file. I made all three changes recommended:

EnableMMAP off #<-- uncomment this line
EnableSendfile off #<-- uncomment this line
Win32DisableAcceptEx #this line had to be added

And this morning, memory use in the server monitor is flatline. Goes up to 200/250 meg, then falls to 150 or so. No long steady climb. Hopefully we won't see any sudden spikes, but that will take a while to make sure.

[mack_]

Rick,

Thanks for taking the time to write the solution once you found it.

Mack