Help With Coldfusion

[Pawaller99]

Hi,

We’re seeing very high CPU utilisation on one of our ColdFusion 11 Update 7 servers running on Windows Server 2012.We are running the server in AWS using an AMI from the Marketplace which includes the ColdFusion license etc.  The instance is an M3 Large, which is 2vCPU and 8GB memory.I have attached a trace of CPU utilisation over the past 3 months.  As you can see, we have very spiky workload, and have had high CPU issues in the past, but if we look at the far right hand side of the graph, you can see it has been pinned at 100% for a few days now:

We have checked out the process monitor on the server, and we can see the WEB-INF file being created and queried over and over.We have settled JVM value min to 512 MB and max to 2048. But that does not solve the issue. We also checkout schedule tasks and looping structure but that seems fine. 
Can you kindly help ? 

[carl type3]

Hi,

Maybe worth patching CF to current for CF11 that would be update 19. Reminder to run WSCONFIG to upgrade tomcat connector.

Any errors or warning in CF-error or CF-out logs?

What version Java would CF11 be running? I guess 8 of sorts so perhaps bring that  to current 1.8.0_251. I don't think CF11 supported Java 11 and fail to recall if it could be fitted that that.

What value does CFadmin > settings > JVM have for Metaspace -XX:MaxMetaspaceSize=?

Wonder why AWS AMI CF images are CF11 not CF2018 since CF11 is end of life.

More questions than answers, Carl

[Dave Watts]

This doesn't really help with the OP's question, but I really don't care much for the ColdFusion AMIs available in the AWS Marketplace. You have to pay a significant fee above and beyond the Windows usage fee, they're not always set up in the best way, and they tend to lag behind official releases. Better to learn how to set up CF and do it yourself, in my opinion.

Windows Server 2012 is also old. It hasn't reached EOL yet, but there's no reason to use it when newer versions of Windows are available.

More on-topic: the WEB-INF directory is just where all the ColdFusion Java binaries live, so you're going to see those in Process Monitor quite a bit.

Dave Watts, Eidolon LLC

[Charlie Arehart]

pawaller, the problem can have many causes, and I share a couple here, including possible solutions. I also suggest a way to get more direct help if you want it.

1) Let's start with one strong possibility given what you've shared: perhaps you're running out of heap within CF, which would cause excessive GCs, which can take high cpu. One way to know is to look for "out of memory" errors referring to "heap" in the coldfusion-error.log. Are they there at the time in question? (With better CF or Java monitoring tools or logs that can be enabled, we could also see if major GCs were happening, taking long, and then KNOW if that was the cause of high CPU.)

Assuming you see such errors in that log at the time, you say you've raised the heap to 2g. Maybe that's not enough (more on why, in a moment). You don't say how much memory is on the box and how much is free at the time of the high cpu. If you had another gig or two free, you could raise the cf max heap by that amoubt (and restart CF). That may or may not be enough, but it's worth a shot.  Time would tell. (If you "still" have issues, be sure to ensure you see the same errors, and if so, consider raising the heap more--but again only within the amount of available memory on the box.)

You would reasonably ask why this problem would be happening "suddenly" and especially if you may say "nothing changed in cf", but it could be increased (perhaps unexpected) traffic, or of a different (perhaps unexpected) nature. Or perhaps you or someone has changed some config related to CF, or in CF, or perhaps code has changed--and maybe that's the root cause, though you may think it "should not be".

That's one idea. And between Carl, Dave, and myself we may well solve it.

2) But sometimes problems like this can't be solved via such back and forth. There are often just too many variables, and too many diagnostics to consider (where different findings can lead in different directions).

We will all try to help as more evidence may be offered, but sometimes there's just no substitute for having someone look at the server directly. And I list folks who offer that at https://www.cf411.com/cftrouble. (Dave and Carl, for some reason I don't see you there. Would you like me to add you? I would certainly want to.)

Note that such help need not require you providing "remote access" to the person assisting but instead could just be a screen share session, as I do with people daily.

I realize that some only prefer to get (or see) free help here and some may really object to anyone even mentioning products or services, even of others. Again, the challenge is just that some problems do end up requiring such direct help, and some people may be willing to hire such help but perhaps didn't know of such alternatives.

3) Finally, if it's not about heap and you don't want direct help, the next recommendation I would make would be to have better monitoring of what's going on inside of CF. You mention being on CF11 on the AWS AMI, and that runs the Enterprise edition. You could perhaps get more useful info from the CF Server Monitor, accessible from within the CF Admin. But folks would warn you not to turn on its "start memory tracking" feature (and perhaps even its "start profiling") as such can add more burden than value.

3a) In fact, you may want to simply check if that "Start memory tracking" especially IS enabled. That alone can contribute to high CPU use (and excessive gc's). You don't need to open the CFSM to see that: you can see the CF Admin "monitoring settings" page, which has 3 checkboxes for the 3 features that can be "started". Are any on? (Don't worry about the 4th checkbox for now. It would not impact CPU.)  If you turn that off, that alone might help.

4) If those are all off, and if the CFSM somehow failed to suffice to identify and resolve the problem, I will point out that FusionReactor would be my next recommendation: it has means to watch (and log) all GCs (number and duration), all java memory spaces (heap, metaspace, and more), as well as tracking things like how many and what requests are running, what queries are running, and much more. Any of those (as well as FR's available alerts) can be key to solving when you have such challenging problems.  (And it can of course be helpful beyond just problem-solving, to include assessing resource usage, trends over time, and much more, including across CF restarts.)  I'll leave it at that, for now.

Either way, hope some of the key CF-specific suggestions I shared may help. It will be great to hear how things resolve.

[Charlie Arehart]

Was there ever a solution, pawaller99? Did any of the the responses below help at all? 

[BKBK]

Hi Pawaller99,

I have good news and bad news. 

First, the bad news. CPU usage regularly reaching 100% can have consequences. Software processes respond differently to resource starvation. Which is what this is.

Some processes may behave in an unforeseen way, while others may produce unexpected results or even fail. Have you checked whether your application has actually been doing what it is expected to do? 

Now, the good news:

1. The 100% CPU has, very likely,  just one cause. The usual suspects are: query, long or recursive loop, large-XML processing and cfhttp.
2. There is a clue to the location of the process that is causing the CPU hike: a scheduled task. In fact, we know more. It runs once daily. To see this, take a look at my interpretation of your picture:

 

[BKBK]

Hi Pawaller99,

Any developments on this issue?