Coldfusion 2018 docker image frequently slow

Thanks for the input @Dave Watts . I am not sure how well CF supports linux in general. It could be one of the libraries but would you know if it its actually not CF but the way httpd works with tomcat ? just grasping at straws really becuse dumping execution timestamp in the application code iteslef shows that the application slows down and takes longer to process .

My other guess is the database layer or hibernate creating these random issues. but once again thats grasping at straws

Thanks @Charlie Arehart and @Dave Watts for your replies

I should clarify a few things 

- I only updated to the latest minor version for java . Did not go to java 12

- please ignore the garbase collection issue. I realised i made a mistake in the settings. However as suggested, updating GC doesnt help

- when i say compilation, it just shows my ignorance of java/CF . all i mean to say is that seems to be loading everything from scratch

For diagnostic , I have fusion reactor installed. all i can tell from that is unusual heap mem usage when its slow 

each dip occurs after I click a page and then makes this saw tooth patern untill the page is loaded. however when the server is working fine i get a simple straight line for heap usage

slightly different when i change the GC to G1 and the server is still slow

Following @Dave Watts suggestion , I installed CF on a fresh EC2 ( amazon linux 2 ) and the exact same thing happenes. The server was responsive and fast for first boot and then slowed down when I restart CF. I did run into a small issue with a few errors but resolved it by setting the heartbeat_interval to 0 as per CF2018 sporadic Crash .

I am fairly comfortable with docker containers and linux though new to Java/CF as mentioned earlier.

Thanks again for any help

My answer will be shorter than @Charlie Arehart 's. It seems like you haven't been able to narrow down the problem very much - this isn't your fault, but basically you're presenting this as "I'm running on Docker and having this problem." I suggest you try a broader set of troubleshooting measures to see if you can find the root cause. What happens if you just run the app on Linux, for testing purposes? Just throw up an EC2 instance in AWS and put it on there, if you can do that - it'll save you a lot of OS configuration stuff. Put your container in a different Docker environment and see what happens.

As for upgrading Java, I recommend upgrading to the latest minor version. This usually isn't listed in the CF docs as a supported Java version, but it's always worked for me. Unfortunately, that doesn't work for major versions. So, for example, if your version of CF supports Java 11.0.14 you can safely upgrade it to 11.0.15 (a minor version) but not to 12 or higher (a major version).

Also, I doubt this has anything to do with compilation. CF actually started as an interpreter, not a compiler, and it doesn't do everything exactly as an ideal compiler would. In fact, you should be able to run CF fine as a single user without compiling anything.

Finally, I have no idea why you wouldn't be able to change the GC algorithm to something other than parallel GC - CF itself does support all kinds of GC options. But you might just want to take a look at the jvm.config and the startup log after you've relaunched in that Docker instance to see if your change picked up.

Dave Watts, Eidolon LLC

Faraz, this is certainly an interesting challenge, and there could be many explanations for it. Before I share any "guesses", I'd ask first:

• though I doubt that changing the Gc algo will necessarily be the solution (it rarely is), there's no reason you can't. So when you say the "the server doesnt start with anything other than" ParallelGC, do you mean the container won't start? That may have more to do with HOW you are changing it. But again I doubt changing it is THE solution. 
• what makes you think it's about compilation? Even if every request WAS being compiled, I'd not expect that to add MINUTES (or even seconds) to a request. Do you have some evidence to suggest that's the issue?

Since it's sometimes slow from the start, even with "no load", that would seem to suggest there's no cf configuration problem (since it works fine "all day" in some cases) 

I really think the best way to solve your issue is to get diagnostics in place, to tell you WHY any one request is slow, as well as what else is going on when this happens.

And sadly, the logs may or may not help in this case. First, have you looked at ALL the cf logs within the container (those recently modified)? As you may know, the docker logs shows only the equivalent of the coldfusion-out.log and coldfusion-error logs, not the other cf logs). Second, in a case like you're describing, the logs often won't help anyway, as some problems don't lead to anything showing in the cf logs--or they don't show what we need to actually diagnose the problem. 

And as you may know, there are indeed various solutions that can help do that (some better than others, depending on the problem), from traditional Java diagnostic tools, to popular Apms (like Newrelic, Datadog, and Dynatrace), to more cf-specific tools like the cf PMT (new since cf2018) or FusionReactor.

All these can be used with containers also, though there can be new challenges doing that, even for folks familiar with the tools (and with running cf in a container). There are also tools specific to your container runtime (Docker) and to the container and host OS which could help, depending on where the problem is (may be cf, may be docker/container config, may be host resource issues). All that's a lot to consider. 

And I'm not saying one has to assess them ALL. But I'm saying that one would investigate a few key diagnostics, and go from there to consider others. 

But if one may be new to BOTH running cf in containers AND using such diagnostic tools, I'd argue that most folks in that situation would have a really tough time going it alone.  There are just so many variables, from container issues to tool use to interpreting what the tools report. Or they may overlook a vital clue. And there's just no way to convey here all one would need to consider.

Then there are questions of you went about setting up your images/containers. There's no "one way", and any one choice could be at issue here. Of course, the fact that things to change for you midstream is another wrinkle that only complicates the diagnosis. 

So while it may be possible that someone else may chime in with JUST the right answer based solely on what you've shared here (or folks may start tossing out guesses of things to try), I'd recommend given all the above that the fastest and most effective way to resolve your specific problem would be to have me (or someone similarly experienced) to join you in a remote screenshare session (like zoom), where together these diagnostics could be considered.

And it may surprise you to hear that I think we could be done within an hour (maybe less, maybe more), since I do this sort of cf troubleshooting daily. And most folks learn a lot as we go. You can learn about my rates, approach, satisfaction guarantee, and more at carehart.org/consulting. I offer there also my online calendar, with slots today and any day this week or coming ones.

I wish I had just "the answer" for you. If you follow these forums, I do usually reply with that or some specific things to consider. There are just far too many variables in this scenario for me to see any to recommend without the diagnostics above. There is likely ONE problem and solution. The challenge is finding it, and I would look forward to helping solve this for you.