CF 23 ColdFusion Launcher Application maxing out CPU

[Madhabdhk]

I am using ColdFusion 2023 - Enterprise 

Every couple of weeks the ColdFusion Launcher Application will go up and max out the CPU on the server.  Site will then begin to timeout.  Restarting the ColdFusion server services is required to resolve the issue. 

The configuration for the server is: 
CF Server: ColdFusion 2023 Enterprises 
RAM: 16 GB
Server: Windows Server 2022 Standard

[carl type3]

Hello,

Anything of interest from coldfusion-out.log or coldfusion-error.log at the time when CPU is maxed out?

Windows or VM tools don't really say much about what is happening in the Java and tomcat layers of CF. How about enable JMX (Java Management Extensions) then use JMC (JDK Mission Control) to see how those internals look. If not keen on free tools then perhaps a paid tool like FusionReactor could help to know more about why.

Regards, Carl.

[Madhabdhk]

Hello Carl,

Thank you for your response.

I've reviewed the coldfusion-out.log and coldfusion-error.log files, but unfortunately, there is nothing of note during the times when the CPU is maxing out. The logs don't seem to indicate any errors or unusual activity that coincides with the spikes.

I haven't enabled JMX or used JDK Mission Control yet, but that sounds like a good next step. I’ll look into enabling JMX to gather more details from the Java and Tomcat layers. I’m also considering FusionReactor as an option if I need more in-depth insights.

I’ll proceed with these suggestions and will update the thread if I find anything interesting through JMX or JMC.

Thanks again for your help!

[BKBK]

My immediate thoughts are the same as those of @carl type3 : the logs. What do the following logs say when the CPU maxes out?

coldfusion-out.log

coldfusion-error.log

server.log

[Madhabdhk]

Hi BKBK, 
Thanks for your response. 

I have reviewed the logs you mentioned—coldfusion-out.log, coldfusion-error.log, and server.log—but unfortunately, I did not find any entries that correspond to the times when the CPU spikes occur. There are no specific error messages or unusual activity recorded during those periods.

[Charlie Arehart]

I'll add a different perspective: I won't be surprised if you find zero info in the logs to explain this. I've seen that many times before, as Carl and BKBK may have.

But my first suspicion (based on your initial evidence) is that Cf may be running out of memory (heap). And while you MAY see outofmemory errors (likely in the coldfusion-error.log), sometimes you won't. Or well before your restart, it may well complain of gc overhead limit reached. That would be a warning that the jvm (underlying cf) is doing excessive Gc's (garbage collection, to remove any old unused objects)--but there is not much it can gc, so heap use remains high and Gc's continue, clogging the cpu and making requests hang. 

And in that case, the new challenge is finding WHAT is holding on to memory. And thus will begin a possibly very long series of posts here as we toss out possible causes and possible tools. 🙂 

But let's not get ahead of ourselves. Maybe you'll find a simpler explanation. A boy can have dreams. 

[Madhabdhk]

Hello Charlie,

Thank you for your perspective and insights.

I agree that it’s not uncommon to find little to no relevant information in the logs during these CPU spikes. Your suspicion regarding memory (heap) issues resonates with me, especially if we’re not seeing clear OutOfMemory errors. The warning about "GC overhead limit reached" could definitely explain the high CPU usage due to excessive garbage collection cycles.

I’ll keep this in mind as I investigate further. Finding out what is holding onto memory will be crucial. I plan to install FusionReactor to gather more information. 

Thanks again for your helpful thoughts. I’ll keep everyone updated as I gather more data.

Best,
Madhab

[Scott_PALADEM]

When you say "Every couple of weeks", is there a precise pattern of every two weeks on a regular interval, or are the intervals varied and just happen to be something around every 2 or three weeks? If it's a regular interval (like every other Wednesday or something) then perhaps that might coincide with some scheduled job that happens every two weeks? 

another thought is that you log requests at the web server level (apache/iis) and see if you can use that to identify which processes might be running or if you are getting traffic surges, etc

[BKBK]

What you're facing appears to be a recurrent issue (occuring every 2 weeks). So here's another idea to explore: possible interference by a security scanner. 

Such a scanner can push ColdFusion into an infinite loop. For an explanation, see https://helpx.adobe.com/coldfusion/kb/coldfusion-performance-issues-troubleshooting.html   

The solution would then be to modify the file {COLDFUSION_HOME}\cfusion\lib\jetty.xml as follows:

replace

<Set name="host">0.0.0.0</Set>

with

<Set name="host">127.0.0.1</Set>

Then restart ColdFusion.

[JJ Frankowski]

I would endorse looking into BKBK's idea about a possible security scan interfering with it. I have previously experienced where one server would have a maxed-out CPU like you show, occuring exactly every morning after a weekly scan. It was a different level of CF, but the only thing that could be done was to restart the CF application via the services.

[Madhabdhk]

Hello everyone,

I wanted to update everyone on our recent efforts. we have increased the RAM from 8 GB to 16 GB, and since making this change, we have not observed any CPU spikes so far.

To give you some context, the issue previously occurred twice a week, specifically on the last Thursday and Friday. It seems that the additional memory may have alleviated the problem, but I’ll continue to monitor the system for any further irregularities. If anything changes, I’ll be sure to share it.

Thank you for your ongoing support!

Madhab

[Charlie Arehart]

Thanks for the updates, Madhab. On this last one, sure, raising the heap MIGHT solve the problem temporarily--or it might even solve it permanently. It could be that you "just needed a bit more" heap than you had. It could be that some combination of factors has contributed to it, whether only recurrently (as you've indicated) or perhaps it's been some recent increase in traffic that's pushed you closer to that limit over time.

If you hit the new limit, it will be time to consider either what the objects are which are piling up, or the traffic causing it, or something else. 

[carl type3]

Hi Madhab,

Likewise good to know for now the heap increase has helped.

Still, could be a good idea to monitor what the heap is doing and that is not difficult or cost money. You could do JMX JMC as mentioned. An alternate option - enabled JVM logging then read the output log file or pass the log file through something EG gcviewer to know results.

Cheers, Carl.

[BKBK]

Hi @Madhabdhk , 

When you say you have "increased the RAM from 8 GB to 16 GB", do you mean you have increased the maximum heap size (Xmx) in ColdFusion from 8 GB to 16 GB? If so, then the situation still needs to be improved.

The Operating System's RAM is 16 GB. If ColdFusion's maximum heap is 100% of the Operating System's RAM, there will again be problems sooner or later. That is because ColdFusion will starve Windows of memory resources.

You can solve this in one of two ways:

1.  Use for ColdFusion a maximum heap size (Xmx) that is lower than 16 GB. Say, a value in the range 12 to 14 GB. However, a Java application may, on rare occasions, use more memory than assigned on the heap. So we're still pushing it a bit here. 
2.   Increase the RAM on Windows to 32 GB or more, and use for ColdFusion a maximum heap size (Xmx) of 16 GB or more. FusionReactor will tell you just how much memory your application needs, to which you can add a margin.

I personally prefer the second option. It is more flexible, sustainable and future-proof.

On the subject of heap size, there is yet another point to consider: setting the initial heap size (-Xms) and the maximum heap size (-Xmx) to the same value, for example,

-Xms16g -Xmx16g
or
-Xms12g -Xmx12g

By default, Java adjusts the heap size dynamically between the (-Xms) and (-Xmx) values, as needed during runtime.  It might be that, on your system, Java does a lot of heap-resizing when the ColdFusion Launcher runs. 

[Madhabdhk]

Hi guys,

Yes, we have increased the RAM on Windows and followed your instructions to set the initial and maximum heap sizes to the same value. 

I’m happy to report that after making these adjustments, we haven’t encountered any issues. I’ll also look into FusionReactor to better understand our memory usage.

Thanks again for the advice,

Madhab

[Charlie Arehart]

Madhab, great to hear things have settled. To be clear though, you'd said in Oct that merely raising the heap had made things much better.

Are you now saying you have had more problems, which have been solved only by raising it further? Or was it raising memory on the box? Or was it setting the heap min=max?

I press for the sake of people who may find this thread in the future, or who are reading it now and might come to conclusions. Just wondering if you know which of the three it was, or if you did all 3? 

[Madhabdhk · Accepted solution]

Hi @Charlie Arehart,

Thanks for following up — great question!

Yes, you're right that initially, just increasing the heap size did help noticeably. However, over time I noticed that performance issues still cropped up intermittently, especially under heavier loads. So I went ahead and made all three changes:

1. Increased the RAM on the box from 8 GB to 16 GB.
2. Increased the default heap size to better utilize the available memory.
3. Set the min and max heap sizes to be equal, which helped reduce GC pauses and improve stability.

After applying all three, the system has been running much more smoothly and consistently. Thanks again for your thoughtful troubleshooting and for keeping the thread clear and useful!

Best regards,
Madhab

[Charlie Arehart]

Thanks for the update and the kind regards. I suspect points 1 and 2 were sufficient. Point 3 is and has long been asserted to be valuable but I've not found it to be nearly as important.

And while sometimes such improvements may be sufficient going forward, other times it's only a stop-gap and the problem returns. In that case, while raising things still further may suffice, generally the next step would be to find and stop whatever is DRIVING the need for more memory. Sometimes it's about unexpected traffic (in nature, volume or both),  sometimes it's about coding or configuration, still other times it's about a bug.

And sometimes Adobe has a bug fix. You're on cf2023, but even with the latest update (from April) there's a known bug causing a memory leak caused by use of caching, including query caching. Ask Adobe for the fix for cf-4244890 (a private bug, not listed on tracker). It's been a solution for other folks I've helped recently. But your issue could have some other cause, of course.

Again, though, perhaps you'll have. I further memory issues. Hope it's so, for your sake. 

[BKBK]

@Madhabdhk , Thanks for the update.