Known Participant

Question

Garbage Collection Overhead Limit & Java Heap Space Issues

Forum|Forum|2 years ago
July 22, 2023
4 replies
2849 views

We're currently running a Windows server with CFM 2018 installed and recently developed a new site. When we push the site to a "live" environment the server runs well for about 20-30 minutes and then it will just fail to respond. I do see the CFM processor and SQL processor slowly start to increase as the site gets traffic. I'm guessing it just builds up until it gets bottlenecked or something. The site then fails to respond or gives us the pretty standard CFM error or fails to load any pages and eventually just throws the Tomcat error. The errors i get are Java heap space and or GC overhead limit.

When we run the site in a "dev" environment it works fine but of course when it's in a dev environment it's only one or two people hitting the site. The "live" and "dev" site are on the same server, they're just different URL's so there is no difference between the server settings. Just the number of people hitting the site (public vs private). I do have a couple of CFC Objects that run a bunch of functions so not sure if I need to investigate there.

If anyone may know of something regarding GC or Java heap please let me know. Any help or direction is appreciated.

Thanks.

B

BiffJinglesAuthor

Known Participant

Ok, three years to almost the day. A similar thing has started happening and nothing has really changed except the increase in traffic to the site. For some magical reason Cold Fusion and SQL Server have recently gone off the rails and have started growing in memory size and eventually CFM going through its motions of GC Overhead issues and not responding. About a month ago things were fine with the Min Heap Size at: 512 and Max Heap Size at 2048. Then CFM started crashing a bit more with GC Overhead issues, so i raised the Min Heap Size to 1024 and Max Heap Size to 4096. For the past few days ColdFusion can't seem to run more than 8 hours without stalling on page timeouts after 30 seconds or GC overhead issues and SQL Server growing in memory size. I haven't added any new code or SQL queries, it's just seen an increase in traffic mostly.
It's a Windows server with 16GB of Ram, ColdFusion 2018 (yah i know it's old).

if this matters, but this is in the JVM Arguments as well.
XX:MaxMetaspaceSize=192m
-XX:+UseParallelGC

I'm totally perplexed... any suggestions would be welcome.

I've attached what Google Analytics is seeing at this moment in time in regards to traffic.

Thank you.

pete_freitag

Participating Frequently

Given the that you had the same problem around the same time of year, and it went away, my guess is that you have some memory heavy code that is only used seasonally.

Look at the web server logs and see if there are any pages that are primarily accessed at this time of year. Then look at the code. You might have a big loop that is adding variables on each iteration, or you might be adding varaibles to a scope such as (session, application, or server) which are not allowed to be garbage collected until they timeout.

Check and see how long your session timeout is - if it is super long (the default is 20 min), then every new visitor or bot request is going to have memory overhead for the duration of the session.

The traffic numbers from google analytics aren't crazy high, most applications would be able to handle that no problem with the amount of heap you have, so that's why I think it is more likley a code or session duration issue. Keep in mind that bot traffic may or may not show up on google analytics.

B

BiffJinglesAuthor

Known Participant

Hello @pete_freitag

Thanks for the reply and suggestions. I looked into your suggestion of looking at the the code and parts of the website that are used. The site is a tourism site so a lot of it gets utilized all year long. Majority of the site that the public hits is for it's event listings and blogs and those pages aren't anything intense. One thing you did bring up is the session timeouts. In my application file i do have the applicationTimeout and sessionTimeout set to 1 hour, could that be too long? You are correct that the hits weren't that intensive on GA, but up to just a few weeks ago, nothing really changed except the amount of traffic. They have started advertising on YouTube, TikTok, Facebook and Google. I just became aware of that after looking at the log files that you and @Charlie Arehart suggested in the next reply (which i'll answer). Let me know if you think my application/session timeouts are too long.

Thanks

Charlie Arehart

Braniac

Biff, our conclusions are the same (Dave and mine): raise the max heap size. Sounds like you could make it 2gb (2048mb, if you like). No need to consider or wonder about anything else, seriously.

As for the maxmetaspacesize, it has nothing to with the heap. If you want to understand it (and why you may not need it at all), I have a blog post on it that Google will readily find.

As for the useparallelgc, that too is the default. Some debate changimg it, but I am 100% confident it's inconsequential for your situation.

As I said, raise the heap and call it a day. If you want to wonder later "why it was needed", we can discuss that after solving the more important problem, which can be so easily solved. 🙂

/Charlie (troubleshooter, carehart. org)

D

Dave Watts

Inspiring

This is kind of a blast from the past. We used to run into similar problems with content-heavy CMS sites, where you had a lot of stuff that didn't change frequently. Anyway, at the time the JVM had three "generations" for objects in the heap: young, tenured and permanent. Things have changed a bit since then, but the basic idea will probably still apply for you. The problem we had was that each generation had its own maximum size, so if we didn't size them correctly we'd run into the same problem. This was also with 32-bit JVMs which were a lot smaller than today's. The way we solved the problem was by first just increasing the total heap size, and if that didn't work we'd find which generation was problematic and increase the size of that generation.

The default heap size in CF used to be 512, which was pretty small. The heap won't get bigger over time, so you need to figure out how big it should be. Or, like me, you could just jack it up to a much larger number if the server can handle that. It can't exceed the available memory of the entire server of course. So, you might go with 1024 or 2048 and see how that works. Even if that doesn't solve the problem, it should take longer to occur and that means you're on the right track. Increase it again until you solve the problem. If you don't have enough memory to do this, get some more and try it again. Memory is pretty cheap.

Dave Watts, Eidolon LLC

B

BiffJinglesAuthor

Known Participant

Thanks @Dave Watts for the reply. The only thing i wasn't sure of was the arguments for JVM. Per my reply to @Charlie Arehart with the JVM settings i wasn't sure what these JVM arguments meant.
XX:MaxMetaspaceSize=192m
-XX:+UseParallelGC

C

carl type3

Braniac

Hello,

I agree with other posts here about increasing initial and maximum size for heap values.

Could be you want to increase -XX:MaxMetaspaceSize=192m or not define it at all letting JVM manage sizing for that. On Java 11 I prefer to set it and define an initial size. Java 17 which is nil CF2018 support offers some JEP fixes which solve Metaspace ballooning when not defined. Perhaps you could try:
-XX:MetaspaceSize=356m -XX:MaxMetaspaceSize=640m

CF2018 default install offers -XX:+UseParallelGC Garbage Collector. The default for Java 11 is -XX:+UseG1GC. I know which I prefer but care needs to be taken. With any recommendation to tune memory spaces or change collector you should do some JVM monitoring to know what the usage is currently like, then what usage is like after altering values.

Regards, Carl.

Charlie Arehart

Braniac

Before you or anyone here gives even a minute of thought about your app or cf itself, please answer this simple question: what is the max heap size in the cf admin "java and jvm" page? It may be 1024, in both prod and dev, and it's merely the default. That may be fine in your dev env, with virtually no traffic, but it may not suffice for prod. Again, don't even worry about contemplating the implications of that.

Next, what is the total memory in the box? And how much is currently shown free? You may only need the max heap to be a LIITTLE more than it is, but if you could double or triple it, you may find all's well. Could you give it ALL the free memory on the box? Maybe, but it isn't always smart.

Finally, if this cf app previously ran in prod on some other cf server, you could also check what the heap size was there.

Let us know how it all goes.

/Charlie (troubleshooter, carehart. org)

B

BiffJinglesAuthor

Known Participant

First off, thank you for the quick response @Charlie Arehart. Greatly appreciated.

The current Minimum JVM heap size is 512MB. Maximum JVM Heap Size is 512MB. A few years ago i fiddled with these settings so not sure if they're relevant. I just used the same settings from when we merged to CF2018 from CF11. Not sure if this matters, but this is in the JVM Arguments as well.
XX:MaxMetaspaceSize=192m
-XX:+UseParallelGC

Current Memory is 8.0 GB, sitting at 3.6. (see attached) Just a note, this is the only site that is running on the server. It's pretty much our last CFM site, the only other program on here is SQL. It's pretty much a dedicated server for this site so i can tweak to it's delight.

The dev site and live site are on the same server so they both are using the exact same settings for CFM. We did migrate from CF11 to CF2018 a few years ago, so i mostly took notes before smoking CF11. Also, not sure if this matters CFM is in lockdown mode or whatever they call it.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded