We recently upgraded to ColdFusion 2018 Enterprise, which is currently running on four different servers. Each of these servers has two instances running in a cluster. The servers are Win 2K19 with 8GB RAM. CPU usage in generally very low (i.e. < 10%). The number of concurrent users would be less than 50, which are accessing a single site, so I would not say there is excessive load.
We have found that each Adobe ColdFusion Launcher Application service will start at around 700-800 MB and then slowly climb up until the point where, presumably, there is no more memory available to it and the ColdFusion site will become unavailable. Restarting the services resolves the issue until it eventually happens again.
I have tried increasing the JVM memory allocation earlier from 1024 MB to 2048 MB in the CF admin, but I am guessing this is just going to allow the system to operate for a bit longer that usual rather than actually fixing the underlying problem.
Any suggestions on how to resolve this would be greatly appreciated as it is making our CF sites very unreliable.
I should have added for reference, two instances on one of the servers are currently consuming 1,350MB and 1,265MB, without a significant increase in overall usage since the last service restart a bit earlier today.
The information you provide is non-specific. So it's difficult to say. Nevertheless, some suggestions:
Beyond BKBK's suggestions, I would note that running 2 CF instances in 8g is pushing it a bit. You also would have a web server on there (probably IIS, right?). Do you have any database running on there? Some do. It's generally recommended to not--but if nothing else the box must be sized appropriately to all that. (And honeslty, most people have more than 8g on their phone. Isn't it something that we often have to run enterprise server apps with fewer resources.)
Anyway, you have reported that the memory will "then slowly climb up until the point where, presumably, there is no more memory available to it and the ColdFusion site will become unavailable." You haven't said what those memory amounts were then, though you added later that "now" they are at about 1.3g. Do you recall what you saw?
And what is your max heap size for CF? That's viewable in the CF Admin, on its "java and jvm" page. And BE CAREFUL TO look at BOTH instances. Each has its own CF Admin. Also, there is likely a cfusion instance on the machine. Is it running also? It has its OWN admin and OWN heap size. What are the 3? And do you see only 2 or all 3 CF services running, now?
Finally, look in the coldfusion-error.log for each instance (each instance has its own folder, as a sibling to cfusion, in your ColdFusion2018 folder). In that coldfusion-error.log for each instance, look at the top of it. When does it start? Assuming it's at least within the past week, do a find on the word outofmemory. Is there any in that log? If so, what does the next line report was the TYPE of oom error? heap? gc overhead limit exceeded? metaspace? Those will be clues. And look for not just the FIRST such error but any that appear preceding such an outage.
Based on what you find, we may have different things to do or to consider further.
There are indeed lots of things to consider in a challenge like this, and each can lead in a new direction. It can be very hard to solve these sort of problems via back and forth here. If you want to "just fix it", there are consultants (like myself) who can help, remotely and perhaps quickly. I list such consultants at cf411.com/cftrouble. But I will try to keep an eye out for other responses here. And other folks may well reply with follow-up ot your answers also.
Thank you for your suggestions, BKBK and Charlie. I'll combine my answers in this post in an effort to cantralise things.
As mentioned, given the memory being consumed does seem to plateau at ~1.3-1.4GB, perhaps that suggests a memory leak is not the problem.
To try and alleviate the problem we implemented a couple of automated restarts of both instace services in an attempt to reset the memory usage. However, we did still have one occassion where ColdFusion became unresponsive only a few hours after one of these restarts, so it seems like they may be ineffective.
I'm a bit stuck on where to go from here other than seeing if the new updates make any difference or whether we can assign some more memory to the server.
Java 11.0.10 and ColdFusion update 11 are currently up-to-date. So we can rule them out. That can only mean that, for your particular set-up, 2 GB RAM is insufficient.
Current ColdFusion versions require a minimum of 2 GB RAM. Adobe recommends 4GB. See ColdFusion Enterprise system requirements.
Suppose then that:
You should use the following practical benchmark: M/N < 0.8
You may use either Performance Monitoring Toolkit or FusionReactor. The choice is 50-50, given your modest set-up and the fact that you're starting out with monitoring.
Performance Monitoring Toolkit is free. FusionReactor service is by paid subscription, after a free trial, but the fee is affordable.
Just so I am clear, whern you say "for your particular setup, 2GB RAM is insufficient", you are referring to the Maximum JVM Memory Heap Size value?
@jeremyf8427004 : Just so I am clear, whern you say "for your particular setup, 2GB RAM is insufficient", you are referring to the Maximum JVM Memory Heap Size value?
Yes. I would strive for a minimum of -Xmx4096m on each instance.
Thank you. We have increased the heap size to 4096 as suggested. Hopefully this helps resolve the problem.
Nice to hear. Please let us know whether your application continued to work as expected.