ColdFusion 2018 Enterprise Memory Consumption

Thank you for your suggestions, BKBK and Charlie. I'll combine my answers in this post in an effort to cantralise things.

BKBK:

• Java version is 11.0.10
• ColdFusion update 11 was applied earlier today on each instance on the server.
• We do not currently use a monitoring tool. I will try to give FusionReactor a try as suggested. Does it have any advantages over Adobe's Performace Monitoring toolkit.

Charlie:

• I take your point regarding the amount of memory available. I will speak with our IT people and see if we can get more added. What would you recommend for a two instance cluster? The server is running IIS, is not hosting any databases, and we have the service for the cfusion instance switched off.
• In terms of memory usage, the instances seem to start at ~700MB following a restart, and then gradually climb to ~1.3-1.4GB where, so far, they do seem to plateau at. I am not sure whether this is typical or a reasonable amount of memory consumption.
• The max heap size for both instances is 2048MB (min: 256MB). Cfusion is the same. I increased this on Monday from 1024MB.
• Only the the services for our two new instances are running (cfusion is switched off).
• Checking the error logs on both instances I cannot find anything in regards to "gc overhead limit exceeded' or 'metaspace'. However, I did find that we had 20 different error files filled with the same error message over and over again - an error with one of the datasources trying to connect. I have fixed this up in the relevant CF page and this error has now stopped.

As mentioned, given the memory being consumed does seem to plateau at ~1.3-1.4GB, perhaps that suggests a memory leak is not the problem.

To try and alleviate the problem we implemented a couple of automated restarts of both instace services in an attempt to reset the memory usage. However, we did still have one occassion where ColdFusion became unresponsive only a few hours after one of these restarts, so it seems like they may be ineffective.

I'm a bit stuck on where to go from here other than seeing if the new updates make any difference or whether we can assign some more memory to the server.

Beyond BKBK's suggestions, I would note that running 2 CF instances in 8g is pushing it a bit. You also would have a web server on there (probably IIS, right?). Do you have any database running on there? Some do. It's generally recommended to not--but if nothing else the box must be sized appropriately to all that. (And honeslty, most people have more than 8g on their phone. Isn't it something that we often have to run enterprise server apps with fewer resources.)

Anyway, you have reported that the memory will "then slowly climb up until the point where, presumably, there is no more memory available to it and the ColdFusion site will become unavailable." You haven't said what those memory amounts were then, though you added later that "now" they are at about 1.3g. Do you recall what you saw?

And what is your max heap size for CF? That's viewable in the CF Admin, on its "java and jvm" page. And BE CAREFUL TO look at BOTH instances. Each has its own CF Admin. Also, there is likely a cfusion instance on the machine. Is it running also? It has its OWN admin and OWN heap size. What are the 3? And do you see only 2 or all 3 CF services running, now?

Finally, look in the coldfusion-error.log for each instance (each instance has its own folder, as a sibling to cfusion, in your ColdFusion2018 folder). In that coldfusion-error.log for each instance, look at the top of it. When does it start? Assuming it's at least within the past week, do a find on the word outofmemory. Is there any in that log? If so, what does the next line report was the TYPE of oom error? heap? gc overhead limit exceeded? metaspace? Those will be clues. And look for not just the FIRST such error but any that appear preceding such an outage. 

Based on what you find, we may have different things to do or to consider further.

There are indeed lots of things to consider in a challenge like this, and each can lead in a new direction. It can be very hard to solve these sort of problems via back and forth here. If you want to "just fix it", there are consultants (like myself) who can help, remotely and perhaps quickly. I list such consultants at cf411.com/cftrouble. But I will try to keep an eye out for other responses here. And other folks may well reply with follow-up ot your answers also.

I should have added for reference, two instances on one of the servers are currently consuming 1,350MB and 1,265MB, without a significant increase in overall usage since the last service restart a bit earlier today.