Approximately one week ago we had an issue with one of our ColdFusion 2018 servers (something along the lines of "could not process Application.cfc"), which required a restart of both instances running on the server. I am not sure if this part is relevant but when restarted the first instance came back fine, while the second would not start for approximately 10 minutes (it took I think three separate service "start" attempts before it eventually came back online).
Prior to the restart we had consistent memory ussage on both instances of ~10-12% (average daily), while it has now been in the 25-40% daily average range for the last week according to FusionReactor. While there were minor changes to three cfm/cfc files on the same day, I don't believe they were anything that would explain this increase. Other metrics like CPU usage appear to be consistent with what they were previously.
Both instances have a minimum JVM heap size of 256MB, and a maximum of 5120MB. The server itself has 24GB total available, with approximately 10GB currently available.
Does anyone know what might explain this increase in memory usage? I am not even sure if the increase is actually an issue or whether the JVM memory management is now just working a bit differently for some reason? I guess my main concern is if there has now been a memory leak introduced, although the daily usage seems to be a bit up and down so I am not sure about that either.
I have attached a screenshot of the Heap Memory Graph for one of the instances from FusionReactor for reference.
Is your FusionReactor running as either its version 9.2.0 or 9.2.1? If so, there was a problem (with its default choice to enable some cf monitoring) which caused cf heap use to rise. Your graph doesn't look that bad, but since your concerned, if you update to 9.2.2, let us know if that solves things.
Thanks Charlie, FusionReactor is currently version 8.7.0, so it seems like we are a bit behind. Would what you described still be applicable?
Nope, it would not. Again, the problem started only with FR 9.2.0.
So as for your issue, I'll say again that your graph does not look worrisome to me, but I realize it does do you. So let's do this: rather than judge just from "how high" the FR heap graph goes (or how high your OS memory for the CF process reportt to be), please go to the Metrics>Web Metrics page in FR, and click the "garbage collection" button below the bottom right CPU graph.
This requests a "major GC" (whereas the frequent drops you see are usually about "minor GCs".) A GC tells the JVM to release any objects that are in that "used" amount but aren't really any longer in use. (We as developers don't tell Java--or CF--to allocate or deallocate memory. That's handled "for us".)
The point is: does that dark orange ("heap used") value drop dramatically? Your graph shows your heap max to be about 4g (the lightest orange). If the dark orange/heap used falls to 1g (half of what was your highpoint of 2g, which is 1/4 that max size), then really I'd not worry about any of this if I was you. You'd have 3g of headroom.
If you feel the highest values of the dark orange were about 1g before and you wish it "stayed there", well that's not how the JVM works. If you set the heap max to 4g, it could get lazy and let memory rise to nearly that max before it would (on its own, then or earlier) do a GC. We don't need to baby-sit the heap--and we DEFINITELY do not to "click the GC button all the time". It's there just for this diagnostic purpose above.
And if your focus is indeed more on the OS (Task Manager on Windows) reporting that the coldfusion process is "using a lot more memory", note that's caused (mostly) by the "allocated memory" (which is the middle orange color). That will rise as heap use rises--and it will fall over time when heap use falls. It will NOT fall just because you do a GC. It could take minutes or longer before the JVM "lets go" of memory that had been marked as "used".
Finally, if you still just have angst about "why is memory use higher" (even if not high enough to really worry), there can be any number of things to cause it. For instance, perhaps some automated requests are being made against your site/app, and such automated requests would each create a new CF session (because they present no cookie, to connect to an existing session). And that would trigger your onsessionstart code in application.cfc (or the equivalent in application.cfm). Perhaps you load up session scope vars, whether with CFC instantiations or queries you want to "cache" for that session, etc.
Note that FusionReactor keeps track of your session count over time (even graphing it) on its
"UEM & Sessions>Sessions" page on the left. See if that may be rising "high" in recent days. If you have not restarted CF in a week, the graph can go back that far so that you can compare over time.
Manually running the garbage collection dropped the used heap to ~250MB on both instances.
Heap max: 4551 MB
Heap allocated: 3438 MB
Heap used: 253 MB
Heap max: 4551 MB
Heap allocated: 3427 MB
Heap used: 251 MB
The session totals seems to be consistent and there are not any new automated processes or similar that would be creating new sessions etc.
Like you have mentioned, there is probably nothing to worry about here, I am just being a bit pathological in that the reported numbers are now different.
Yep, some people would kill to have CF run at only 250mb used heap. Count your blessings and call it a day. 🙂 Hope that the info I shared may help you in the future, or others as they may find it.
The Heap Memory Graph you've shared looks fine to me. However, you should rule out any errors. To do so, have a look at the ColdFusion logs and FusionReactor logs. Also have a look at the Longest Transactions in FusionReactor.
You say there were code changes in cfm/cfc files. That could have led to the increase in memory use. For example, if the changes introduced new objects, memory use would increase.
On to the problem of restarting. The onApplicationStart eventhandler may be using a lot more heap memory in the creation of objects. That might be compounded by the fact that the initial heap size of 256 MB is too small. To verify this idea, set Xms equal to Xmx:
Then restart the ColdFusion instance.
Thank you, BKBK. I have checked the error log and the only issue appears to be the following, but they are also recorded prior to when the restart occurred, so may not be having an impact in regards to this:
SEVERE: Manager [localhost#]: Unable to receive message through TCP channel
I understand this is an issue with session messages being received out of order? Probably a topic for a separate discussion.
In regards to new code changes, there was nothing that I would expect to have a noticable increase in memory - e.g. new obects etc. I will review again to confirm, however.
Finally, thank you for the advice regarding the lower heap size value in regards to restarting. I will take a look at this and see if it makes a difference.