PROBLEMS WITH REQUEST TUNING CF2021

[José38529857rmgv]

Dear

I have a server with Windows Server 2012 R2 and a CF 2021 in production, we have a problem with the request tunning configured on the server(240) that sometimes arrives to the maximum configured and the service goes down. Sometimes is necesary a restart the service for continues with the service.

The Datasource PostgreSQL configured on the client is configureded to Restrict connections to 210.
We had the same configurations implemented on a CF 2016 and a service of monitoring without problems.
The update of CF2021 is the version 13, I attach a snapshot about that.
We activate a service of monitoring FusionReactor and the requests goes up until double or more of connections, so the system is very loaded and blocks easier.

So we don't know what's happening we attach some snapshot about cfstat, FusionReactor and request tunning configured.

I hope you can help me
Best regards.

 

 

[Charlie Arehart]

Jose, I'm afraid that nothing in your message or screenshots here will help us help you to resolve the problem. There's no single obvious setting change, and there's not even clarity about what requests were running. You show the fr cloud display of finished requests. That's not what's running at the time.  Even if you might assert those are ones which at least completed WHILE your cfstat shows many running, there were clearly many more running at the time than those you show. And in fact there may be ones running FAR longer than those, which we don't see here.

In any case, you'd want to drill into the details of those you show, to find out WHY they were slow. I gather that's not something you feel competent to assess--and that's understandable. Most folks would not. 

But I'll say that I do that with people daily. Again, there may not be any need of tuning cf or your code: until you find WHY requests are slow, you can't solve it. And there could be any of many reasons for the slowness.

This is where I could help. It's far too much to discuss here, telling you what to look at, how to understand it, how to connect that to possible solutions. I could write an entire book on this (and practically have across all the blog posts, presentations, and forum thread replies here.) 

But if instead we meet (online, in a  consulting session), I might help you identify and resolve the problem in as little as an hour, and I'd explain things along the way to a) identify and resolve your specific issues(s) and b) help you better understand FR to leverage it more effectively on your own, for use with whatever next problems may arise. If that interests you, find more on my rates, approach, satisfaction guarantee, online calendar, and more at carehart.org/consulting.

Or you could wait to see if someone might have some darts to throw at the dartboard or might want to lead you on a treasure hunt here. My sense is that neither would be especially productive. But if you really want your problem solved, I'd look forward to helping directly. Then you could report back here as to what was the problem and solution, if you like. 

[José38529857rmgv]

Thanks for your response Charlie.

To add some more information to what I explained previously, we previously had Coldfusion 2016, we switched to Coldfusion 2021 approximately 1 month ago, the first problem we had is that if we have the PMT active, the service completely fails after a few minutes ; Reviewing information in the forums, we noticed that fusion reactor is highly recommended as an alternative to PMT, we tried that program, and exactly the same thing happened to us. Product that if we activate any monitoring tool the service fails, we do not have any monitoring tool other than cfstat. When the service fails, as can be seen in the attached screenshot, you can see that the request running reaches the limit that we have configured. At the database performance level (postgresql), our DBA tells us that in some cases the idle connections to the database are triggered, but this does not always happen. The behavior that can be seen in the capture of the request running, this happens whether any monitoring system is active or not, but if we activate PMT or fusion reactor, this behavior causes the service to fail constantly, now at least it happens but to a lesser extent. measured within the day.

[Charlie Arehart]

I'm afraid your words here don't get us to any better identification of or resolution to the problem. We still don't even know what it is. And the images you shared are not any sort of "capture of the request". They're the same sort as yesterday.

And as for your feeling that adding any monitoring only makes things worse, that's an interesting possibility but I suspect there's more to that than meets the eye. 

In any case, I stand by what I said previously: if you want the problem solved, I'm confident I can help you solve it in a remote session togetherz today or tomorrow. To be clear, there's no reason to concern yourself with "impacting production" while we'd work, so don't presume we could only meet in off hours--though we can arrange that if somehow the hours offered in my online calendar don't suit you. FWIW, I'm on US central time. 

[Dave Watts]

I'm going to throw some darts here. First, what happens if you turn off your monitoring service for a while?

Second, how did you get to those numbers for CF request tuning and database connection limits? They presumably came from somewhere. The CF request tuning limit seems awfully high to me. I would spend some time with a load test tool and see why those numbers are as high as they are.

Third, what happens when you try to optimize the slowest page as best you can? What part of that page is slower than the rest? You should be able to use CFTIMER to break the page into separate parts and see what's going on. My guess is that your database query is really slow when it presumably shouldn't be (I'm not going out on much of a limb there) but at least it's a start. Take that query and use whatever query analyzer PostgreSQL has, so you can figure out why it might get slow under load (no indexes, bad indexes, poorly-written SQL, poorly-designed DB structure for INSERT/UPDATE, poorly-designed DB structure for SELECT). This is an area where @Charlie Arehart or someone else can probably help you quickly.

Dave Watts, Eidolon LLC

[Romina376147489sje]

Hello Dave, thanks for answer.  My name is Romina, and I work on the same team as Pedro, so I'll be addressing your questions.

Firstly, the system operates smoothly when we do not use the monitoring service.

Secondly, those numbers come from our experience. We've been using them for many years and have adjusted them based on server responses. However, we will review the CF request tuning limit since you mentioned it seems quite high. Is there a method you know to calculate it that isn't empirical?

Thirdly, the pages do not show slowness, and the system functions well when we do not activate CF Reactor. We've spent the last three years optimizing queries, indices, etc., so I don't believe the issue lies there. We also recently validated this with an external DBA.

Thank you very much for your assistance and your responses.

Best regards,
Romina

[Romina376147489sje]

Dave, I apologize. In my response, I said Pedro, but I meant Jose.

[BKBK]

@Hi @Romina376147489sje , thanks for the new information. If you and your team answer the other questions, you will likely get solutions from the forum.

If water is pouring into a tank faster than the taps are running, an overflow will be the result. That is how I currently see it.

[Dave Watts]

Hi, Romina, Jose & Pedro!

Unfortunately, I guess, there's no calculation to find the optimal value for a given tuning value. That's why load testing is so important. You change the value, you test it, you see the result, and you repeat until you find the optimal value. This can take weeks or months.

In the meantime, you might not want to use PMT or Fusion Reactor if they're causing problems. Maybe, once you've optimized your system the old-fashioned way (load testing) you can use those tools for daily performance monitoring etc.

Dave Watts, Eidolon LLC 

[BKBK]

@José38529857rmgv ,

It is clear that your ColdFusion server and application need tuning. First, some questions I have, then some suggestions.

Questions:

(1) You seem to be using the US date format. In which time-zone is your ColdFusion server? And FusionReactor?

(2) Your application is apparently memory-intensive. So what are the server RAM and the values in the java.args property in the jvm.config file of the ColdFusion application?

(3) What is the value of "Queue Timeout Settings" (in the ColdFusion Administrator)?

(4) You had problems installing both ColdFusion's Performance Monitoring Toolset (PMT) and FusionReactor.  (You say, "if we activate any monitoring tool the service fails"). These are two independent applications.. That suggests there is likely a fault in your ColdFusion installation. What errors or issues (in detail) did you encounter when you tried to install PMT and FusionReactor? 

(5) Does your application use Application.cfc? If so what are the values for:
this.sessionManagement

this.clientManagement
this.clientStorage
this.sessionTimeout
this.setClientCookies
this.setDomainCookies

(6) Apparently, your application passes CFID and CFTOKEN in the URL. Is that how session is maintained in the application? Or is session maintained in a different way?

(7) The application seems to receive an excessive number of requests per second. What is the average number of simultaneous users? Are any of them machines that send automatic requests to the application?

Suggestions:

(a) To understand why the application worked smoothly with ColdFusion 2016, but not with ColdFusion 2021, you have to compare like with like. ColdFusion 2016 had all the required packages installed. So the first thing to do is to make sure all the packages in ColdFusion 2021 are installed. The steps to do so are as follows:

1.  Open the command prompt (CMD) as administrator;
2.  Use the DOS command CD to navigate to C:\ColdFusion2021\cfusion\bin;
3.  Type cfpm.bat and press ENTER. That should bring up ColdFusion's Package Manager tool.
4.  Type install all and press ENTER. When ColdFusion finishes, type quit and press ENTER.
5.  Close the CMD window.
6.  Restart ColdFusion 2021. 

(b) Your application is apparently data-intensive and makes frequent connections to the database. Where the input remains the same, use cachedWithin to cache queries. That will significantly improve performance.

(c) The application seems to read and write an excessive amount of data per second. You could improve performance by

1.  caching frequently requested (unchanging) request and response data; 

2.  enabling compression for data transfer, to reduce the amount of data sent over the network;

3.  applying load balancing;

4.  using asynchronous processing in your code, for example, by means of threads;

5.  making use of the services of a Content Delivery Network to serve static files, thereby reducing the load on the web server.

[Charlie Arehart]

Can you share how you were able to conclude that "Your application is apparently memory-intensive"? It's certainly one of many common problems. I could have listed still more. I'm asking if you could help us all to see what suggested that specifically. 

As for the rest, such back and forth reviewing these general things offered MAY luck out and strike gold, but the issue may well be still something else entirely. There are ways using fusionreactor (which he had already) to get that, but again going into detail here could get voluminous. Then there's his assertion of trouble upon using such tools (which should be resolvable).

Jose, you never replied to my (or Dave's) responses. If you'll reply to bkbk's, I hope you'll at least offer some response to ours to let us know what you think. In the silence, we can infer anything.

Finally, if you just want this problem solved, perhaps in less than an hour, my offer of a remote consulting session stands. Some challenges are hard to solve in forum threads but are far simpler in an online session--where we target your issue rather than elaborate back and forth here in a bunch of guesses, suggestions, explanations, questions, answers, clarifications, etc. Your call, of course.

(And before anyone may carp back that I'm "always driving people to consulting to get answers", that's absolutely not true. By far in most of my replies here I make no mention of it. And in those I do, it's when I think it may help. Only rarely do I stress that it's  in my opinion the MOST effective option, as in this case.) 

All 3 of us (Dave, BKBK, and me) are frequent contributors here. Often we're on the same page, sometimes we differ in opinions. Either way, we all do really just want to help. 

[José38529857rmgv]

Good day.

Before starting, I wanted to apologize for not responding earlier.

I would like to comment that we have managed to stabilize the platform by making adjustments to the simultaneous Template requests parameter, we had it at 325 and now we set it to 350; we also made adjustments to the workers.properties and server.xml files, modifying the following values:

workers.properties

- worker.cfusion.max_reuse_connections: 1200

- worker.cfusion.connection_pool_size: 8400

server.xml

- maxThreads: 8400

Pic1

Pic2

Pic3

As you can see in the screenshots, the tweaks helped keep the platform stable but currently we can only monitor using cfstat via a DOS window. Despite the apparent stability, we have episodes that can vary between 4 to 30 seconds where the active requests shoot up to the maximum and reach the maximum value set for the server, a screenshot of this is attached. (Pic2)

We have times where active requests hang for more than 60 seconds, causing the entire system to not respond, we even lose access to the CF administrator portal, when this situation occurs the only way to correct it is to restart the CF service.

The installation process of CF, PMT and FusionReactor was carried out without problems, however when PMT or FusionReactor are activated, the active requests begin to increase until they reach the configured limit, but this happens much faster and more often than when PMT or FusionReactor are not active.

As mentioned before, we currently do not have any monitors active for our CF servers, the PMT and FusionReactor services are stopped and disabled, and if we activate them, the active requests are triggered and generate a service crash that forces us to restart the CF service.

Now I'm going to answer the questions that I have been asked by BKBK, Dave and Charlie.

Q: Second, how did you come up with those numbers for the CF request fit and database connection limits? Presumably they came from somewhere. The CF request adjustment limit seems to me terribly high. I would spend some time with a load test tool and see why those numbers are as high as they are.

A: As Romina mentions, these parameters have been adjusted thanks to the experience of working with versions CF 8, CF 2016 and now CF 2021, in fact as can be seen above, we managed to stabilize the platform much more than it was at the worst moment of the incident.

Q: It seems that you are using the US date format. In which time zone is your ColdFusion server located? And FusionReactor?

A: MM/DD/YYYY is the format that you are using CF and FusionReactor with the timezone GMT-4, this configuration is the default and we have not modified CF or FusionReactor.

Q: Your application is apparently memory intensive. So what is the server RAM and values in the java.args property in the jvm.config file of the ColdFusion application?

A: The server has 256GB of RAM installed and has between 16GB and 64GB of RAM allocated, as seen in the parameters below

java.args=-server  -Xms16384m -Xmx61440m --add-opens=java.rmi/sun.rmi.transport=ALL-UNNAMED --add-opens=java.base/java.nio=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/sun.util.cldr=ALL-UNNAMED --add-opens=java.base/sun.util.locale.provider=ALL-UNNAMED --add-opens=java.base/java.net=ALL-UNNAMED --add-opens=java.base/java.io=ALL-UNNAMED -XX:+UseParallelGC -Djdk.attach.allowAttachSelf=true -Dcoldfusion.home={application.home} -Duser.language=en -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.JavaUtilLog -Djava.util.logging.config.file={application.home}/lib/logging.properties -Dtika.config=tika-config.xml -Djava.locale.providers=COMPAT,SPI -Dsun.font.layoutengine=icu -Dcom.sun.media.jai.disableMediaLib=true -Dhttps.protocols=TLSv1.2 -Dcoldfusion.cpu.count=8 -Dcoldfusion.searchimplicitscopes=true -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib/,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/cfform/jars,{application.home}/bin/cf-osgicli.jar

Q: What is the value of "Queue Timeout Settings" (in the ColdFusion Administrator)?

A: 560 seconds; this value was defined by the dev team, this is because we have processes that take a long time to be executed, according to what the dev team indicates this delay is normal. If for example, we lower this value to 60 seconds, timeout errors begin to be reported by system users.

Q:Does your application use Application.cfc? If so what are the values for:

A:

sessionmanagement="Yes"

clientmanagement="yes"

sessiontimeout="#CreateTimeSpan(0,6,0,0)#"

setclientcookies="true"

the other values is undefined

Q:Apparently, your application passes CFID and CFTOKEN in the URL. Is that how session is maintained in the application? Or is session maintained in a different way?

A: Precisely, we keep the variables CFID and CFTOKEN in the URL

Q:The application seems to receive an excessive number of requests per second. What is the average number of simultaneous users? Are any of them machines that send automatic requests to the application?

A: The number of concurrent users occupying the system varies depending on the time of use and the platform, but the average maximum is 250 users per second and there's no an automatic machine that sends requests.

Q:Your application is apparently memory-intensive?

A: AVIS is not a memory-intensive application

Thank you very much for your assistance and your responses.

Best regards

[BKBK]

Hi @José38529857rmgv , thanks for answering our questions and for the additional information. I can already see possible ways to fine-tune the application. However, I have to finish my own project off before the end of this week.

Expect suggestions from me this weekend. 

[José38529857rmgv]

Hi @BKBK  thanks for your response, Hoping you are well, we will be attentive to your comments

Regards

[BKBK]

Here, as promised, are some thoughts and suggestions.

1.    Number of ColdFusion instances
Assumed: 1 (If 2 or more, please say how many.)

2.   Settings in workers.properties and server.xml

workers.properties: I am not too sure about the use of a colon in the workers.properties file. I would therefore suggest that you use the equal sign (=) instead of the colon (:).

In addition, be careful how you set the value of the properties max_reuse_connections and connection_pool_size. Setting either of them too high or too low, can lead to a reduction in throughput. The excessive connection_pool_size value of 8400 may therefore be suboptimal. (In any case, the value of max_reuse_connections may never exceed that of connection_pool_size.)

server.xml: The total value of connection_pool_size (in workers.properties) may never exceed the value of maxThreads (in server.xml). In addition, the values of connectionTimeout (in server.xml) and connection_pool_timeout (in workers.properties) should match. However, note that connectionTimeout is in milliseconds, whereas connection_pool_timeout is in seconds.

So, for optimal performance, ensure that :

• the values of max_reuse_connections , connection_pool_size and maxThreads are equal.
• the value of connectionTimeout (in milliseconds) equals the value of connection_pool_timeout (in seconds).

2.1   workers.properties
worker.cfusion.max_reuse_connections=1200
worker.cfusion.connection_pool_size:=1200
worker.cfusion.connection_pool_timeout:=60

2.2   server.xml
<Connector protocol="AJP/1.3" maxThreads="1200" connectionTimeout ="60000"> </Connector>

References,
https://www.petefreitag.com/blog/tuning-tomcat-iis-connectors/ 
https://coldfusion.adobe.com/2014/05/coldfusion-11-iis-connector-tuning/

https://christierney.com/2014/12/09/max_reuse_connections-conundrum-coldfusion-11-tomcat-iis-connect...

3.   ColdFusion Administrator settings
Maximum number of simultaneous Template requests: Cfstat shows that the total number of current requests at any time (Reqs Q'ed + Reqs Run'g) is 426. So, I would raise the Administrator setting, "Maximum number of simultaneous Template requests" to, say, 450.

Timeout requests waiting in queue after ...seconds: You say that requests that take longer than 60 seconds "hang". So, let's suppose that you expect each request to take at most one minute.

In that case, the setting "Timeout requests waiting in queue after 560 seconds" is probably excessive. You should lower it to, say, "Timeout requests waiting in queue after 120 seconds".

Timeout Requests after seconds: The Req TO'ed column in cfstat shows that there are 9 persistent long-running requests. They are a likely cause of poor performance, as long requests can hog server resources. To improve performance, configure ColdFusion to time out requests that take longer than 60 seconds., say.

To do so, go to the page Server Settings > Settings in the Administrator, and set the value of "Timeout Requests after seconds" to 60. Remember to press the button to "Submit Changes".

4.   ColdFusion Administrator datasource and PostgreSQL settings
Take a look at the Pg/Sec and DB/Sec columns in the cfstat table. They show that:

• the ColdFusion application frequently accesses the PostgreSQL database thousands of times per second..
• each page in the application accesses the database on average 9 times.

You could therefore improve performance by fine-tuning access between the ColdFusion application and the database. Make sure ColdFusion pools datasource connections. You can do so in the ColdFusion Administrator., for each and every datasource used by the application.

4.1   ColdFusion Administrator datasource

Maintain Connections: Click on the button to edit the datasource. Click on "Show Advanced Settings", select "Maintain Connections", then click on "Submit Changes".

4.2  PostgreSQL

max_connections: Increase the value of max_connections in PostgreSQL from the current 210 to, say, 300.

shared_buffers: It is usual to also increase the value of shared_buffers proportionally. In this case by a factor of 3/2. (The default value of shared_buffers is 128 MB).

Change the two PostgreSQL settings, for example, from

max_connections = 210
shared_buffers = 128MB

to

max_connections = 300
shared_buffers = 192MB

References:
https://stackoverflow.com/questions/30778015/how-to-increase-the-max-connections-in-postgres 
https://wiki.postgresql.org/wiki/Tuning_Your_PostgreSQL_Server 

5.   Java settings
coldfusion.cpu.count: What is the JVM setting -Dcoldfusion.cpu.count=8 for?
The setting apparently interferes with CPU. As such, it is performance-sensitive at server level.

Unless you have a good reason for using it, I would strongly suggest that you delete it.

Memory use and choice of Garbage Collector(GC): My initial guess was that "Your application is apparently memory-intensive ". I said so because the application frequently accesses the database at a rate of thousands of times per second, which can be memory-intensive.

Anyway, when I said that, I didn't know your heap-size settings, which are -Xms16384m -Xmx61440m. These settings alone confirm the description of "memory-intensive".

Choice between G1GC and ParallelGC: As the heap size (Xmx) is well over 4 GB, your application will likely perform better with the G1GC garbage collector than with ParallelGC. 

Refer to the attached PDF, in which Chat GPT compares G1GC with ParallelGC. It points out the other factors to consider besides heap size

Therefore, switch from

-XX:+UseParallelGC

to

-XX:+UseG1GC

Xmx and Xms: The effect of the settings Xms and Xmx is to get the Java Virtual Machine (JVM) to start initially with Xms amount of memory and be able to use a maximum of Xmx amount of memory. For single-server applications in production, it is customary to set the same value for Xms and Xmx. The fixed heap size means the JVM won't spend time recalculating the old and new generation sizes.

Reference:
https://developer.jboss.org/thread/149559 
https://docs.oracle.com/cd/E19159-01/819-3681/abeik/index.html 

So, use, for example:

-Xms64g -Xmx64g ,
-Xms32g -Xmx32g ,
or even
-Xms16g -Xmx16g

You should be able to justify your choice. FusionReactor (Ultimate Edition) will show you how much heap size the application needs.

[BKBK]

This is a follow-up to my last post. Here I'll show how to determine the heap size (Xmx, Xms) of your ColdFusion application. There is a simple way to do so without FusionReactor or Coldfusion's Performance Monitoring Toolset. Namely, VisualVM.

The steps in using VisualVM to determine the heap size of a ColdFusion application:

1.  Download VisualVM. The current version is 2.1.9 and the download is the zip file visualvm_219.zip.
2.  Unzip the file on the server on which ColdFusion is installed.  For example, if ColdFusion is installed on the C: drive on Windows, then  you can copy the unpacked folder to C:\visualvm_219. 
3.  Establish a connection between your ColdFusion application and VisualVM. You can do so as follows: (a) Open ColdFusion's configuration file, jvm.config, in an editor; (b) Add the following 3 flags to the java.args property: -Dcom.sun.management.jmxremote.port=8701 -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
4.  Save the file jvm.config and restart ColdFusion. When ColdFusion restarts, it will internally be connected to VisualVM on port 8701. 
5.  Launch the executable visualvm.exe by double-clicking. In my example, the file is located at C:\visualvm_219\bin\visualvm.exe.
   You should then see a screen like this one:
   
6.  Right-click on Local, and select Add JMX Connection.
7.  Enter the port number 8701, check the Display Name checkbox and  type the name ColdFusion.
   
8.  When you click on OK, the ColdFusion application will be registered on VisualVM.
   
9.  Double-click on ColdFusion. Then click on the Monitor tab and on the Heap tab.
   You should then see your application's memory usage, amongst others. Hover with the mouse on the display, to get a feel.
   In my example, the heap size is 8.2 GB, of which 2.2 GB is currently being used.
   

[Dave Watts]

I've read through all those numbers, and I can't point out much that's clearly wrong - but it's definitely giving off a bad code smell. One thing I can clearly point out is this:

Q: What is the value of "Queue Timeout Settings" (in the ColdFusion Administrator)?

A: 560 seconds; this value was defined by the dev team, this is because we have processes that take a long time to be executed, according to what the dev team indicates this delay is normal. If for example, we lower this value to 60 seconds, timeout errors begin to be reported by system users.

560 seconds is a REALLY HIGH value for any global timeout setting. If I ran into this as a load tester, and I didn't know anything else about the code, I'd try to identify the specific reasons why it was originally set this high. If a page or module takes this long to execute, why is that? Is it happening across the board, or are some pages and modules taking this long while most aren't? What do those pages and modules share in common? Are they interacting with specific database tables, or performing operations that are known to be long-running like PDF generation, or calculating reports that are more complex than the rest of the application? I dunno, but I'd look for that. If you could set a long timeout just for these pages, and a short default timeout for everything else, that would probably improve performance and reduce memory use significantly.

Here's another potential bad practice.

Q:Does your application use Application.cfc? If so what are the values for:

A:

sessionmanagement="Yes"

clientmanagement="yes"

sessiontimeout="#CreateTimeSpan(0,6,0,0)#"

setclientcookies="true"

That is a REALLY LONG session timeout - SIX HOURS! A session is normally kept for thirty minutes or less, the lower the better. Unfortunately, HTTP doesn't have a built-in session tracker so you have to guess how long users will wait around before clicking a button or a link, etc. Sessions are stored in memory, and this is going to use a lot more memory than required. If a user comes to your site, does something in the first hour, then closes the browser window, your server will still keep that session around for another five hours! So, what you might think of as not memory-intensive may turn out to unnecessarily hold onto memory it doesn't need to. Try setting this to thirty minutes or less in a test environment, and see what happens. If your developers say "we need it to be this long because of blah blah blah" that's a good sign you need new developers.

The rest of my objections are just code smells like I mentioned before, but this application seems to need a LOT more memory than, well, nearly any other CF app I've seen. I'd be really interested to see how much memory you actually need. You shouldn't need a 64GB max heap size unless there's some heavy-duty stuff going on. I really doubt you'd have 250 concurrent users if you had reasonably short sessions.

I'd probably disagree with or at least question some of the other advice you've received. For example, if you increased the min heap size to 64GB as well, that's going to take a while to restart - my wild guess is twenty minutes or more. That's why you get two different values to set. If your max heap isn't that large, or if you have a cluster of web and/or application servers, it will often make sense to set the min heap the same so there isn't so much reallocation going on after startup, but if your max heap IS that large and you have a single application server, you probably don't want all that allocation going on before the machine can start working. But that's my opinion and just a minor quibble. I would definitely start reducing the actual memory allocation needed during the app's runtime. Performance analysis load testing will be your friend here, unless you like developing on production. (Seriously, don't develop on production.)

Finally, and this is sort of unrelated, you want to stop using searchImplicitScopes=true as quickly as possible. This is a critical security issue. Yes, you will have to go through your entire codebase to fix this. It's a giant pain. But there's some help out there! Pete Freitag's Fixinator tool can find and fix these unscoped variables, and Adobe has released a patch that will log these variables. You can find out all about that by scrolling to the bottom of this page:

https://www.petefreitag.com/blog/cf-searchimplicitscopes/

Dave Watts, Eidolon LLC

[BKBK]

I'd probably disagree with or at least question some of the other advice you've received. For example, if you increased the min heap size to 64GB as well, that's going to take a while to restart - my wild guess is twenty minutes or more. That's why you get two different values to set.

By @Dave Watts

Setting Xms lower than Xmx has its own problems. If the application used a large heap size of, say, 64GB, then setting Xms lower than Xmx could lead to heap resizing and unpredictable performance,  In other words, it may save minutes at restart, but cost a cumulative performance drop of hours per day.

If an application requires a large heap size, then it is quite common to set Xms and Xmx to the same value. Some even call it a best-practice. But I would call it a rule-of-thumb, to be used only when you have no other considerations to go by.

With large heaps, setting Xmx and Xms to the same value will likely improve performance. Especially in production environments, where you want to avoid the overhead of a constantly resized heap.  Naturally, the optimal values of Xms and Xmx will ultimately depend on the memory requirements of the application. Hence the need for careful monitoring and testing to find the optimal settings for the specific application and environment.

[BKBK]

Hello @José38529857rmgv and @Romina376147489sje ,

How are you getting on? Any joy, given all the suggestions you have received?