Help With Coldfusion

Hi Pawaller99,

I have good news and bad news. 

First, the bad news. CPU usage regularly reaching 100% can have consequences. Software processes respond differently to resource starvation. Which is what this is.

Some processes may behave in an unforeseen way, while others may produce unexpected results or even fail. Have you checked whether your application has actually been doing what it is expected to do? 

Now, the good news:

1. The 100% CPU has, very likely,  just one cause. The usual suspects are: query, long or recursive loop, large-XML processing and cfhttp.
2. There is a clue to the location of the process that is causing the CPU hike: a scheduled task. In fact, we know more. It runs once daily. To see this, take a look at my interpretation of your picture:

Was there ever a solution, pawaller99? Did any of the the responses below help at all? 

pawaller, the problem can have many causes, and I share a couple here, including possible solutions. I also suggest a way to get more direct help if you want it.

1) Let's start with one strong possibility given what you've shared: perhaps you're running out of heap within CF, which would cause excessive GCs, which can take high cpu. One way to know is to look for "out of memory" errors referring to "heap" in the coldfusion-error.log. Are they there at the time in question? (With better CF or Java monitoring tools or logs that can be enabled, we could also see if major GCs were happening, taking long, and then KNOW if that was the cause of high CPU.)

Assuming you see such errors in that log at the time, you say you've raised the heap to 2g. Maybe that's not enough (more on why, in a moment). You don't say how much memory is on the box and how much is free at the time of the high cpu. If you had another gig or two free, you could raise the cf max heap by that amoubt (and restart CF). That may or may not be enough, but it's worth a shot.  Time would tell. (If you "still" have issues, be sure to ensure you see the same errors, and if so, consider raising the heap more--but again only within the amount of available memory on the box.)

You would reasonably ask why this problem would be happening "suddenly" and especially if you may say "nothing changed in cf", but it could be increased (perhaps unexpected) traffic, or of a different (perhaps unexpected) nature. Or perhaps you or someone has changed some config related to CF, or in CF, or perhaps code has changed--and maybe that's the root cause, though you may think it "should not be".

That's one idea. And between Carl, Dave, and myself we may well solve it.

2) But sometimes problems like this can't be solved via such back and forth. There are often just too many variables, and too many diagnostics to consider (where different findings can lead in different directions).

We will all try to help as more evidence may be offered, but sometimes there's just no substitute for having someone look at the server directly. And I list folks who offer that at https://www.cf411.com/cftrouble. (Dave and Carl, for some reason I don't see you there. Would you like me to add you? I would certainly want to.)

Note that such help need not require you providing "remote access" to the person assisting but instead could just be a screen share session, as I do with people daily.

I realize that some only prefer to get (or see) free help here and some may really object to anyone even mentioning products or services, even of others. Again, the challenge is just that some problems do end up requiring such direct help, and some people may be willing to hire such help but perhaps didn't know of such alternatives.

3) Finally, if it's not about heap and you don't want direct help, the next recommendation I would make would be to have better monitoring of what's going on inside of CF. You mention being on CF11 on the AWS AMI, and that runs the Enterprise edition. You could perhaps get more useful info from the CF Server Monitor, accessible from within the CF Admin. But folks would warn you not to turn on its "start memory tracking" feature (and perhaps even its "start profiling") as such can add more burden than value.

3a) In fact, you may want to simply check if that "Start memory tracking" especially IS enabled. That alone can contribute to high CPU use (and excessive gc's). You don't need to open the CFSM to see that: you can see the CF Admin "monitoring settings" page, which has 3 checkboxes for the 3 features that can be "started". Are any on? (Don't worry about the 4th checkbox for now. It would not impact CPU.)  If you turn that off, that alone might help.

4) If those are all off, and if the CFSM somehow failed to suffice to identify and resolve the problem, I will point out that FusionReactor would be my next recommendation: it has means to watch (and log) all GCs (number and duration), all java memory spaces (heap, metaspace, and more), as well as tracking things like how many and what requests are running, what queries are running, and much more. Any of those (as well as FR's available alerts) can be key to solving when you have such challenging problems.  (And it can of course be helpful beyond just problem-solving, to include assessing resource usage, trends over time, and much more, including across CF restarts.)  I'll leave it at that, for now.

Either way, hope some of the key CF-specific suggestions I shared may help. It will be great to hear how things resolve.

Hi,

Maybe worth patching CF to current for CF11 that would be update 19. Reminder to run WSCONFIG to upgrade tomcat connector.

Any errors or warning in CF-error or CF-out logs?

What version Java would CF11 be running? I guess 8 of sorts so perhaps bring that  to current 1.8.0_251. I don't think CF11 supported Java 11 and fail to recall if it could be fitted that that.

What value does CFadmin > settings > JVM have for Metaspace -XX:MaxMetaspaceSize=?

Wonder why AWS AMI CF images are CF11 not CF2018 since CF11 is end of life.

More questions than answers, Carl