ColdFusion 2018 Enterprise Memory Consumption

[jeremyf8427004]

We recently upgraded to ColdFusion 2018 Enterprise, which is currently running on four different servers. Each of these servers has two instances running in a cluster. The servers are Win 2K19 with 8GB RAM. CPU usage in generally very low (i.e. < 10%). The number of concurrent users would be less than 50, which are accessing a single site, so I would not say there is excessive load.

We have found that each Adobe ColdFusion Launcher Application service will start at around 700-800 MB and then slowly climb up until the point where, presumably, there is no more memory available to it and the ColdFusion site will become unavailable. Restarting the services resolves the issue until it eventually happens again.

I have tried increasing the JVM memory allocation earlier from 1024 MB to 2048 MB in the CF admin, but I am guessing this is just going to allow the system to operate for a bit longer that usual rather than actually fixing the underlying problem.

Any suggestions on how to resolve this would be greatly appreciated as it is making our CF sites very unreliable. 

[jeremyf8427004]

I should have added for reference, two instances on one of the servers are currently consuming 1,350MB and 1,265MB, without a significant increase in overall usage since the last service restart a bit earlier today.

[BKBK]

The information you provide is non-specific. So it's difficult to say. Nevertheless, some suggestions:

• Are you using an appropriate Java version? I would suggest Java SE 11.0.10 (LTS).
• Some memory-related problems in CF2018 have been fixed. So make sure you install the latest update, which is 11. ColdFusion updates are cumulative.
• Do you use a monitoring tool? If so, it will tell you where the memory is going. If not, you should give FusionReactor a try.
• If you don't have the time or the resources, then do, for each instance, a set of thread dumps. Say 3, with a 10-second interval in between. Search the dumps and identify the threads in running or blocked state.

[Charlie Arehart]

Beyond BKBK's suggestions, I would note that running 2 CF instances in 8g is pushing it a bit. You also would have a web server on there (probably IIS, right?). Do you have any database running on there? Some do. It's generally recommended to not--but if nothing else the box must be sized appropriately to all that. (And honeslty, most people have more than 8g on their phone. Isn't it something that we often have to run enterprise server apps with fewer resources.)

Anyway, you have reported that the memory will "then slowly climb up until the point where, presumably, there is no more memory available to it and the ColdFusion site will become unavailable." You haven't said what those memory amounts were then, though you added later that "now" they are at about 1.3g. Do you recall what you saw?

And what is your max heap size for CF? That's viewable in the CF Admin, on its "java and jvm" page. And BE CAREFUL TO look at BOTH instances. Each has its own CF Admin. Also, there is likely a cfusion instance on the machine. Is it running also? It has its OWN admin and OWN heap size. What are the 3? And do you see only 2 or all 3 CF services running, now?

Finally, look in the coldfusion-error.log for each instance (each instance has its own folder, as a sibling to cfusion, in your ColdFusion2018 folder). In that coldfusion-error.log for each instance, look at the top of it. When does it start? Assuming it's at least within the past week, do a find on the word outofmemory. Is there any in that log? If so, what does the next line report was the TYPE of oom error? heap? gc overhead limit exceeded? metaspace? Those will be clues. And look for not just the FIRST such error but any that appear preceding such an outage. 

Based on what you find, we may have different things to do or to consider further.

There are indeed lots of things to consider in a challenge like this, and each can lead in a new direction. It can be very hard to solve these sort of problems via back and forth here. If you want to "just fix it", there are consultants (like myself) who can help, remotely and perhaps quickly. I list such consultants at cf411.com/cftrouble. But I will try to keep an eye out for other responses here. And other folks may well reply with follow-up ot your answers also.

[jeremyf8427004]

Thank you for your suggestions, BKBK and Charlie. I'll combine my answers in this post in an effort to cantralise things.

BKBK:

• Java version is 11.0.10
• ColdFusion update 11 was applied earlier today on each instance on the server.
• We do not currently use a monitoring tool. I will try to give FusionReactor a try as suggested. Does it have any advantages over Adobe's Performace Monitoring toolkit.

Charlie:

• I take your point regarding the amount of memory available. I will speak with our IT people and see if we can get more added. What would you recommend for a two instance cluster? The server is running IIS, is not hosting any databases, and we have the service for the cfusion instance switched off.
• In terms of memory usage, the instances seem to start at ~700MB following a restart, and then gradually climb to ~1.3-1.4GB where, so far, they do seem to plateau at. I am not sure whether this is typical or a reasonable amount of memory consumption.
• The max heap size for both instances is 2048MB (min: 256MB). Cfusion is the same. I increased this on Monday from 1024MB.
• Only the the services for our two new instances are running (cfusion is switched off).
• Checking the error logs on both instances I cannot find anything in regards to "gc overhead limit exceeded' or 'metaspace'. However, I did find that we had 20 different error files filled with the same error message over and over again - an error with one of the datasources trying to connect. I have fixed this up in the relevant CF page and this error has now stopped.

As mentioned, given the memory being consumed does seem to plateau at ~1.3-1.4GB, perhaps that suggests a memory leak is not the problem.

To try and alleviate the problem we implemented a couple of automated restarts of both instace services in an attempt to reset the memory usage. However, we did still have one occassion where ColdFusion became unresponsive only a few hours after one of these restarts, so it seems like they may be ineffective.

I'm a bit stuck on where to go from here other than seeing if the new updates make any difference or whether we can assign some more memory to the server.

[BKBK]

Java 11.0.10 and ColdFusion update 11 are currently up-to-date. So we can rule them out. That can only mean that, for your particular set-up, 2 GB RAM is insufficient.

Current ColdFusion versions require a minimum of  2 GB RAM. Adobe recommends 4GB. See ColdFusion Enterprise system requirements.

Suppose then that:

• your server has N gigabyte RAM
• each of your instances has an -Xmx setting (in jvm.config) of at least 4GB
• the sum of the -Xmx values of all the instances on the server is M gigabyte

You should use the following practical benchmark:  M/N < 0.8   

You may use either Performance Monitoring Toolkit or FusionReactor. The choice is 50-50, given your modest set-up and the fact that you're starting out with monitoring. 

Performance Monitoring Toolkit is free. FusionReactor service is by paid subscription, after a free trial, but the fee is affordable. 

[jeremyf8427004]

Just so I am clear, whern you say "for your particular setup, 2GB RAM is insufficient", you are referring to the Maximum JVM Memory Heap Size value?

[BKBK]

@jeremyf8427004 : Just so I am clear, whern you say "for your particular setup, 2GB RAM is insufficient", you are referring to the Maximum JVM Memory Heap Size value?

Yes. I would strive for a minimum of -Xmx4096m on each instance.

[jeremyf8427004]

Thank you. We have increased the heap size to 4096 as suggested. Hopefully this helps resolve the problem.

[BKBK]

Nice to hear. Please let us know whether your application continued to work as expected.