Ever since we upgraded from ColdFusion 2021 to ColdFusion 2023 we have been dealing with out of memory issues. ColdFusion will run fine for roughly 24-30 hours, then we will start seeing CPU spikes to 100% every 30 seconds. Garbage collection can't free up enough memory so ColdFusion eventually crashes and we have to restart the server.
Things we have tried that don't seem to help:
- Downgrading to 17.0.11
- Tweaking the min and max heap sizes
- Tweaking the caching settings
- Changing the garbage collector algorithm to G1GC
- Tweaking our websites to cache queries for a shorter period of time (1 hour down to 15 minutes down to 5 minutes)
Here are our current settings:
Min Heap: 8192
Max Heap: 8192
Garbage Collector: UseParallelGC
Cached Templates: 1000
Cached Queries: 5000
We do have Fusion Reactor installed on all of our servers but this is like trying to find a needle in a haystack. I really don't know what I should be looking at.
Here is a most recent screenshot from 2 days ago that shows the ventual demise on one of our servers.
I am really at my wit's end here. If this isn't a memory leak I don't know what the heck it is. If anyone has any recommendations on what to try next I would appreciate it.
100
Replies
In your opinion, what scenario is better?
- Allowing an invalid URL parameter to throw a cfqueryparam error (Current situation)
- Stripping the invalid value before it gets to the database and using cfthrow to throw a manual error
- Stripping the invalid value before it gets to the database and do not throw any sort of error
By @davecordes
Making a connection to the database and executing a query are expensive processes. Does the application know who is making the URL request? If not, I wouldn't advise a trip to the database.
I would therefore do it as follows:
<!---
It is assumed that the user has been validated.
That is, the application has code for a boolean such as isValidUser.
--->
<cfif isValidUser and isNumeric(URL.text)>
<cfquery>
select *
from tbl_test
where id=<cfqueryparam cfsqktype="CF_SQL_INTEGER" value="URL.test">
</cfquery>
<cfelseif isValidUser and not isNumeric(URL.text)>
<!--- Ensure that an appropriate error page is displayed to the user --->
<cfelse>
<!--- Ignore --->
<!--- If necessary, log details of requester, for example, IP, location, visit frequency, etc., for forensic purposes --->
</cfif>
Let's revisit the heap histogram from FusionReactor:
The classes coldFusion.sql.InParameter and java.lang.Integer led us to cfqueryparam errors, which in turn led us to bots.. Given what you've just discovered about the impact of bot traffic, I think there is more where that came from.
Errors are transient. So you wouldn't expect ColdFusion to maintain objects pertaining to errors in memory. The one thing pertaining to cfqueryparam that makes sense for ColdFusion to maintain in memory is a query. In other words, a cached query.
Therefore, we could hypothesize that those coldFusion.sql.InParameter and java.lang.Integer objects on the heap represent cached queries containing cfqueryparam of integer cfsqltype.
Suppose this hypothesis is correct. Then drastically reducing bot traffic - in essence, the number of cached queries currently in memory and bot access to cached-query code - will lead to a reduction of the frequency of coldFusion.sql.InParameter and java.lang.Integer objects on the heap. Hence, to a reduction of memory usage.
Needless to add, it makes sense to review the cachedWithin attribute of every cached query. Assign to each the smallest feasible time-span value. In fact, only cache where necessary.
I understand what you are saying and certainly @davecordes and @Charlie Arehart have also understood your point.
By @Scott_PALADEM
That wasn't clear to me. You and Charlie, in particular, presumed that a steady session count of 10 000 could not result in an increase in memory. Look back at the posts.
In a scenario where the application is adding additional data to the session as the user navigates the app like you are suggesting, I personally would expect to see the memory usage on the server to have more peaks and valleys throughout the day as various sessions start and end.
By @Scott_PALADEM
I agree with you. However, I wondered what would happen if sessions never ended. Hence my suggestion that Dave look into this.
I think that everyone understands everyone's suggestions and there is no need to keep belaboring your point.
By @Scott_PALADEM
Isn't it because I belaboured the point that you now understand it?
I understood your points from your first comment.. I simply don't think that session data is most likely cause of the symptoms I am seeing.
if what you're suggesting was the issue then this graph would look completely different:
You will note that 30 minutes into this chart, the sessions are being destroyed after the 30 minute timeout. There is then a steady level of sessions being created and destroyed.
if there were 10k active sessions that were being maintained by bots for 24 hours. Then there would be less orange and a lot more green on this chart.
@Scott_PALADEM , Take another look at the displays:
Your reading of the graphs at the top chimes with mine. Sessions are indeed being destroyed about as fast as they are being created.
However, whereas your focus is on the topmost graphs, mine is on the ones below. That is, the blue. I also took into account the table underneath.
The first ColdFusion instance created, on average, 22757 sessions per hour and destroyed, on average, 25667 sessions per hour. The second ColdFusion instance created, on average, 22735 sessions per hour and destroyed, on average, 22394 sessions per hour.
With 200 simultaneous users and 8000 daily users, I wondered how the 10 000 active sessions arose and what their impact might be.
Dave has offered some clarification in his recent post. If I understand it correctly, we have to read that as 5 000 active J2EE sessions. Even so, I am still at a loss to know what's going on. I hope my cfdump suggestion will provide a more definitive clue.
To be honest, I have no answers on this subject ("sessions"), only questions. That explains why my list of suggestions was extensive.
AdChoices