PROBLEMS WITH REQUEST TUNING CF2021

[José38529857rmgv]

Dear

I have a server with Windows Server 2012 R2 and a CF 2021 in production, we have a problem with the request tunning configured on the server(240) that sometimes arrives to the maximum configured and the service goes down. Sometimes is necesary a restart the service for continues with the service.

The Datasource PostgreSQL configured on the client is configureded to Restrict connections to 210.
We had the same configurations implemented on a CF 2016 and a service of monitoring without problems.
The update of CF2021 is the version 13, I attach a snapshot about that.
We activate a service of monitoring FusionReactor and the requests goes up until double or more of connections, so the system is very loaded and blocks easier.

So we don't know what's happening we attach some snapshot about cfstat, FusionReactor and request tunning configured.

I hope you can help me
Best regards.

 

 

[Charlie Arehart]

Jose, I'm afraid that nothing in your message or screenshots here will help us help you to resolve the problem. There's no single obvious setting change, and there's not even clarity about what requests were running. You show the fr cloud display of finished requests. That's not what's running at the time.  Even if you might assert those are ones which at least completed WHILE your cfstat shows many running, there were clearly many more running at the time than those you show. And in fact there may be ones running FAR longer than those, which we don't see here.

In any case, you'd want to drill into the details of those you show, to find out WHY they were slow. I gather that's not something you feel competent to assess--and that's understandable. Most folks would not. 

But I'll say that I do that with people daily. Again, there may not be any need of tuning cf or your code: until you find WHY requests are slow, you can't solve it. And there could be any of many reasons for the slowness.

This is where I could help. It's far too much to discuss here, telling you what to look at, how to understand it, how to connect that to possible solutions. I could write an entire book on this (and practically have across all the blog posts, presentations, and forum thread replies here.) 

But if instead we meet (online, in a  consulting session), I might help you identify and resolve the problem in as little as an hour, and I'd explain things along the way to a) identify and resolve your specific issues(s) and b) help you better understand FR to leverage it more effectively on your own, for use with whatever next problems may arise. If that interests you, find more on my rates, approach, satisfaction guarantee, online calendar, and more at carehart.org/consulting.

Or you could wait to see if someone might have some darts to throw at the dartboard or might want to lead you on a treasure hunt here. My sense is that neither would be especially productive. But if you really want your problem solved, I'd look forward to helping directly. Then you could report back here as to what was the problem and solution, if you like. 

[José38529857rmgv]

Thanks for your response Charlie.

To add some more information to what I explained previously, we previously had Coldfusion 2016, we switched to Coldfusion 2021 approximately 1 month ago, the first problem we had is that if we have the PMT active, the service completely fails after a few minutes ; Reviewing information in the forums, we noticed that fusion reactor is highly recommended as an alternative to PMT, we tried that program, and exactly the same thing happened to us. Product that if we activate any monitoring tool the service fails, we do not have any monitoring tool other than cfstat. When the service fails, as can be seen in the attached screenshot, you can see that the request running reaches the limit that we have configured. At the database performance level (postgresql), our DBA tells us that in some cases the idle connections to the database are triggered, but this does not always happen. The behavior that can be seen in the capture of the request running, this happens whether any monitoring system is active or not, but if we activate PMT or fusion reactor, this behavior causes the service to fail constantly, now at least it happens but to a lesser extent. measured within the day.

[Charlie Arehart]

I'm afraid your words here don't get us to any better identification of or resolution to the problem. We still don't even know what it is. And the images you shared are not any sort of "capture of the request". They're the same sort as yesterday.

And as for your feeling that adding any monitoring only makes things worse, that's an interesting possibility but I suspect there's more to that than meets the eye. 

In any case, I stand by what I said previously: if you want the problem solved, I'm confident I can help you solve it in a remote session togetherz today or tomorrow. To be clear, there's no reason to concern yourself with "impacting production" while we'd work, so don't presume we could only meet in off hours--though we can arrange that if somehow the hours offered in my online calendar don't suit you. FWIW, I'm on US central time. 

[Dave Watts]

I'm going to throw some darts here. First, what happens if you turn off your monitoring service for a while?

Second, how did you get to those numbers for CF request tuning and database connection limits? They presumably came from somewhere. The CF request tuning limit seems awfully high to me. I would spend some time with a load test tool and see why those numbers are as high as they are.

Third, what happens when you try to optimize the slowest page as best you can? What part of that page is slower than the rest? You should be able to use CFTIMER to break the page into separate parts and see what's going on. My guess is that your database query is really slow when it presumably shouldn't be (I'm not going out on much of a limb there) but at least it's a start. Take that query and use whatever query analyzer PostgreSQL has, so you can figure out why it might get slow under load (no indexes, bad indexes, poorly-written SQL, poorly-designed DB structure for INSERT/UPDATE, poorly-designed DB structure for SELECT). This is an area where @Charlie Arehart or someone else can probably help you quickly.

Dave Watts, Eidolon LLC

[Romina376147489sje]

Hello Dave, thanks for answer.  My name is Romina, and I work on the same team as Pedro, so I'll be addressing your questions.

Firstly, the system operates smoothly when we do not use the monitoring service.

Secondly, those numbers come from our experience. We've been using them for many years and have adjusted them based on server responses. However, we will review the CF request tuning limit since you mentioned it seems quite high. Is there a method you know to calculate it that isn't empirical?

Thirdly, the pages do not show slowness, and the system functions well when we do not activate CF Reactor. We've spent the last three years optimizing queries, indices, etc., so I don't believe the issue lies there. We also recently validated this with an external DBA.

Thank you very much for your assistance and your responses.

Best regards,
Romina

[Romina376147489sje]

Dave, I apologize. In my response, I said Pedro, but I meant Jose.

[BKBK]

@Hi @Romina376147489sje , thanks for the new information. If you and your team answer the other questions, you will likely get solutions from the forum.

If water is pouring into a tank faster than the taps are running, an overflow will be the result. That is how I currently see it.

[Dave Watts]

Hi, Romina, Jose & Pedro!

Unfortunately, I guess, there's no calculation to find the optimal value for a given tuning value. That's why load testing is so important. You change the value, you test it, you see the result, and you repeat until you find the optimal value. This can take weeks or months.

In the meantime, you might not want to use PMT or Fusion Reactor if they're causing problems. Maybe, once you've optimized your system the old-fashioned way (load testing) you can use those tools for daily performance monitoring etc.

Dave Watts, Eidolon LLC 

[BKBK]

@José38529857rmgv ,

It is clear that your ColdFusion server and application need tuning. First, some questions I have, then some suggestions.

Questions:

(1) You seem to be using the US date format. In which time-zone is your ColdFusion server? And FusionReactor?

(2) Your application is apparently memory-intensive. So what are the server RAM and the values in the java.args property in the jvm.config file of the ColdFusion application?

(3) What is the value of "Queue Timeout Settings" (in the ColdFusion Administrator)?

(4) You had problems installing both ColdFusion's Performance Monitoring Toolset (PMT) and FusionReactor.  (You say, "if we activate any monitoring tool the service fails"). These are two independent applications.. That suggests there is likely a fault in your ColdFusion installation. What errors or issues (in detail) did you encounter when you tried to install PMT and FusionReactor? 

(5) Does your application use Application.cfc? If so what are the values for:
this.sessionManagement

this.clientManagement
this.clientStorage
this.sessionTimeout
this.setClientCookies
this.setDomainCookies

(6) Apparently, your application passes CFID and CFTOKEN in the URL. Is that how session is maintained in the application? Or is session maintained in a different way?

(7) The application seems to receive an excessive number of requests per second. What is the average number of simultaneous users? Are any of them machines that send automatic requests to the application?

Suggestions:

(a) To understand why the application worked smoothly with ColdFusion 2016, but not with ColdFusion 2021, you have to compare like with like. ColdFusion 2016 had all the required packages installed. So the first thing to do is to make sure all the packages in ColdFusion 2021 are installed. The steps to do so are as follows:

1.  Open the command prompt (CMD) as administrator;
2.  Use the DOS command CD to navigate to C:\ColdFusion2021\cfusion\bin;
3.  Type cfpm.bat and press ENTER. That should bring up ColdFusion's Package Manager tool.
4.  Type install all and press ENTER. When ColdFusion finishes, type quit and press ENTER.
5.  Close the CMD window.
6.  Restart ColdFusion 2021. 

(b) Your application is apparently data-intensive and makes frequent connections to the database. Where the input remains the same, use cachedWithin to cache queries. That will significantly improve performance.

(c) The application seems to read and write an excessive amount of data per second. You could improve performance by

1.  caching frequently requested (unchanging) request and response data; 

2.  enabling compression for data transfer, to reduce the amount of data sent over the network;

3.  applying load balancing;

4.  using asynchronous processing in your code, for example, by means of threads;

5.  making use of the services of a Content Delivery Network to serve static files, thereby reducing the load on the web server.

[Charlie Arehart]

Can you share how you were able to conclude that "Your application is apparently memory-intensive"? It's certainly one of many common problems. I could have listed still more. I'm asking if you could help us all to see what suggested that specifically. 

As for the rest, such back and forth reviewing these general things offered MAY luck out and strike gold, but the issue may well be still something else entirely. There are ways using fusionreactor (which he had already) to get that, but again going into detail here could get voluminous. Then there's his assertion of trouble upon using such tools (which should be resolvable).

Jose, you never replied to my (or Dave's) responses. If you'll reply to bkbk's, I hope you'll at least offer some response to ours to let us know what you think. In the silence, we can infer anything.

Finally, if you just want this problem solved, perhaps in less than an hour, my offer of a remote consulting session stands. Some challenges are hard to solve in forum threads but are far simpler in an online session--where we target your issue rather than elaborate back and forth here in a bunch of guesses, suggestions, explanations, questions, answers, clarifications, etc. Your call, of course.

(And before anyone may carp back that I'm "always driving people to consulting to get answers", that's absolutely not true. By far in most of my replies here I make no mention of it. And in those I do, it's when I think it may help. Only rarely do I stress that it's  in my opinion the MOST effective option, as in this case.) 

All 3 of us (Dave, BKBK, and me) are frequent contributors here. Often we're on the same page, sometimes we differ in opinions. Either way, we all do really just want to help.