Session variables being overwritten when using Redis for session storage
Copy link to clipboard
Copied
Would greatly appreciate if someone could comment on this snippet from the external session storage doc:
Once a session object is received from Redis, there can be changes to the object within the timespan of a given request. The session is persisted (if modified) back to external storage on request end. The changes made by the current request on one node are available to all other nodes.
https://helpx.adobe.com/coldfusion/using/external-session-storage.html
As part of my organization's move to a cluster of cf server instances, we're also moving to storing sessions in a redis cache. Some places in the app make remote invocations via fetch request to first modify the session in some way, then before that request has finished, re-fetch some data which may or may not use the newly modified state. This was fine while we were storing session in memory, but we're finding that the modified state is getting overwritten by the second request (perhaps because that second request pulls from the cache before it was modified by the first request). This is all happening on 1 server using the cache, as we haven't moved to the cluster yet.
Ideally, we'd make 1 single request that does both, but going through our huge legacy app to find where this might happen would be a nightmare. I'm wondering if our situation is something other people have seen--because the documentation makes it seem like the modified session should be available to both requests?
Copy link to clipboard
Copied
First, (since you've asked) I'll offer that I've not heard of anyone hitting this, meaning who's using the app design you use and then switches to redis sessions.
But second, I'd argue that the quote from the docs is specifically speaking TO that concern you raise, in their clarifying that the session changes in the current request are not written back to redis until "request end". And it makes sense: otherwise cf would need to be writing to (and reading from) redis somehow THROUGHOUT the entire request, which would be expensive/chatty. 🙂
I can't fathom what can resolve this, as far as any feature in cf.
(Edit after original post:) Well, there is the older "session Replication" feature, which cf has longer offered-- though configurable in the cf admin only as part of cf's clustering feature, which you don't mention using. (FWIW, the session replication CAN be made to work without that CF clustering feature--and even in cf Standard--by editing files instead of using the CF admin. That's not documented anywhere I can readily point to.) But the session replication is itself very chatty, which made some not care for it, among other reasons.
Maybe others will have better ideas for you.
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
Thanks for the input, Charlie--and you're probably right that reading and writing to redis throughout the request would be expensive. Part of what I was hoping (perhaps beyond reason haha) was that during request 2, coldfusion would recognize that the user's session was already pulled during request 1 and use the session state from temporary memory that was modified in request 1.
Perhaps I misinterpreted the last part of that paragraph, "The changes made by the current request on one node are available to all other nodes." This is probably referring to multiple cf server instances accessing the same redis cache, rather than multiple requests accessing the changes made mid-request.
Copy link to clipboard
Copied
Glad to have helped. And right: I'd read that quote also as just them helping folks connect the dots that a given app running in multiple cf instances or servers CAN share sessions using this redis feature, if they all point to the same redis server.
But again there may well be some way to get what you want. Since Adobe has not chimed in, you may want to ask them at cfsup@adobe.com. If you learn anything new, do let us know! 🙂
(As for bkbk's reply below, I'll leave you to consider and respond to it.)
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
I have been scratching my head on this one. I thought it was straightforward. But your question suggests otherwise. So let's see if I understand your question.
User's session is in progress. User makes request 1, during which the value of session.myVar is changed from session.myVar=50 to session.myVar=5000. User then makes request 2, during which the session scope is dumped.
Do you mean that session.myVar is currently 50 instead of 5000?
If so, two possible causes could be:
- Request 1 had not yet ended, hence the cached session had not been updated.
- Sticky Sessions have not been disabled.
Copy link to clipboard
Copied
If those were the causes, then the corresponding solutions would be:
- Place within a session-scoped lock any code that writes to or reads from a session. For example,
<cflock scope="session" timeout="30" type="exclusive"> <cfset session.myVar = getIt()> </cflock> <cflock scope="session" timeout="10" type="readonly"> <cfset updatedValue = updatedValue + session.myVar> </cflock>​
- Disable "Sticky Sessions" and "Session Replication". I would then restart the instances, just to be sure.
Copy link to clipboard
Copied
Thanks for the help--I don't believe sticky sessions/session replication is pertinent in this current case, because we are still only running on one server.
In my previous debugging, I was able to log session.myVar in the application.onRequestEnd hook. E.g., if session.myVar is initialized as 0 and the first request should set session.myVar to 1, the first log of session.myVar is 1 (the variable was set correctly), but the second log of session.myVar is 0. I had interpreted this to be the first cause you had listed--the first request had not finished before the second one had started and by the time the second is finished, the session state has changed. It appears that the requests only share session variables through redis, so the updated variable will only be available to request 2 once it has written to redis. In other words, "The changes made by the current request on one node are available to all other nodes" ...but changes are not available to other requests on the same node.
I'm rereading "The session is persisted (if modified) back to external storage on request end." and I'm curious on how coldfusion determines if the session has been modified--session.myVar was not directly modified by the second request, but my guess is that, since the value of myVar is different, it will persist and overwrite myVar in the redis cache. I don't think cflock will help because I don't modify session.myVar at all in request 2--but the value has changed since the request started.
Copy link to clipboard
Copied
With just one instance, it is, of course, not mandatory to disable sticky-sessions/session-replication. Still, it is recommended to disable them even when using one node. That said, there is a reason for my suggestion.
We have an issue of which we don't yet know the cause. I suggested sticky-sessions/session-replication only as a possible cause. That would be relevant, for example, if you had eliminated every other possibility. In which case, we would then have to check whether there is a sticky-sessions/session-replication bug. There was one some years ago.
I agree with your conclusions on how ColdFusion handles sessions in this scenario. With one exception. The following statement is not always true: "...the requests only share session variables through redis, so the updated variable will only be available to request 2 once it has written to redis."
An example. Let's say request 1 starts with a session.myVar value of 0, which it changes to 1. This is analogous to a "dirty write" in database terms. Though the updated variable is in memory, it has not yet been saved in Redis. Hence, it may happen that request 2 starts with a session.myVar value of 0, overwriting request 1's update before that update is saved. What I have just described is a race condition.
Hence the need for session-scoped locks: to avoid race conditions. With such locks in place, ColdFusion will handle the process very much as it would a transaction.
My guess is that ColdFusion does something like the following:
- within exclusive lock L1: request 1 starts with a session.myVar value of 0, which it changes to 1. If request 1 is still active, then that is a dirty write.
- within read-only or exclusive lock L2: request 2 starts, but cannot yet read from or write to session.myVar. That is because request 1 continues to hold an exclusive lock on session.myVar. So ColdFusion makes request 2 wait.
- request 1 ends, which signals to ColdFusion to persist session.myVar in Redis.
- request 2 can now read or update session.myVar, assuming that the elapsed time is within the timeout period of lock L2.
Copy link to clipboard
Copied
Guys, I can't help but want to step in with some info for you to consider. I'd stayed out of this subthread until now.
1) First, there's no concept of sticky sessions with the cf Redis sessions. Bkbk, you originally proposed:
Disable "Sticky Sessions" and "Session Replication".
Those are not related to cf redis session at all. They are features of the cf enterprise clustering feature, which Zachary has not said he's using. Are you Zachary? (You may be using some other means to cluster the instances.)
Typically people use cf's redis sessions either because they're on cf standard (which does not offer this cf clustering or session replication feature), or they prefer redis INSTEAD of it (as I'd addressed in my original post, with a tweak added minutes after first posting the reply).
2) Moving on to your cflock suggestion, I can't see that to be at all the solution here. I DO NOT believe that Cf will somehow implement a session-based lock contention across the two cf instances. How could it? The only thing joining them is redis itself. And it could no more enforce such "immediate" contention than it could more simply just do the update in question.
Bkbk, please see the earlier subthread between me and Zachary. He'd replied to that yesterday. (I appreciate he didn't yet mark it as the answer.)
I don't think there's going to be any better solution. But if there is one, I just say here now why I think these two ideas (sticky sessions and cflock) are NOT going to get us there.
3) Finally, I fully realize this could fire up your indignation, bkbk. That's why I'd held off commenting from the first reply you offered. Since Zachary has now engaged with you and you are both proceeding further down this path, I offer this info so that you may consider "turning back".
But you both have free agency. You can choose to dismiss me, or of course can work to prove me wrong. As always, I really am just trying to help. We just tend to present contending opinions on some things (though thankfully sometimes we do agree on other solutions). As long as we remain civil, some people might even benefit from the discourse. I'll plead now that you just respond to the facts presented in points 1 and 2. I'm certainly open to correction where I'm wrong on those.
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
Charlie, thanks for your response. You give me more credit than I deserve, as you assume that I spoke with certainty. I didn't. At the beginning I didn't even understand the issue Zachary had described. 🙂
As I have said, I don't know for sure what the cause is. That is why I suggested a number of possible causes. For the purposes of elimination. That is because I include the possibility that the issue might be caused by a bug.
In any case, let me answer your questions in turn.
- You say, "there's no concept of sticky sessions with the cf Redis sessions" and "Those are not related to cf redis session at all". I agree and didn't say otherwise.
What I wondered was whether Zachary is using Redis with sticky-sessions/session-replication enabled. Hence my suggestion to confirm that those settings are disabled. - You say, "I DO NOT believe that Cf will somehow implement a session-based lock contention across the two cf instances." . The issue at hand involves just one ColdFusion instance. That is what makes it a hard puzzle. To me, anyway.
- The discussion between Zachary, you and me has been good-natured and instructive. I therefore see no need for you to appeal for civility. In any case, rest assured, there is no indignation here.
The issue Zachary has found is at once odd, unexpected and puzzling. In fact, I consider it to potentially be an important ColdFusion issue. If this happens with one user on one ColdFusion instance, what will happen when the architecture is scaled up to a cluster of 7 instances? And when there are thousands of users?
I think it behooves us as a community to get to the bottom of this issue.
Copy link to clipboard
Copied
Thanks, and I have some new perspective to add, which I hope may still be helpful.
1) First let me say that I have been operating under a mistaken assumption. Zacahry's original post opened with "As part of my organization's move to a cluster of cf server instances", and I'd offered my replies to reflect that.
Only now do I notice his clarification within that Op that he was as yet testing things only on a single instance. Perhaps he didn't correct my statements, if he presumed I'd seen that. I see he also stated in in reply to you, but I missed it with that blinder on. Sorry for my confusion.
2) All that said, I'll assert again there's no connection between redis sessions and session replication/sticky sessions. There won't even be such a setting to tweak if one has not created a cf cluster (and again which one CANNOT even do if running cf standard). Zachary, which edition are you running (standard or enterprise, or trial or developer), just to get that question clear.
3) As for cflock, since this Is a single instance, my previous concern no longer applies (about how cf would propagate that to another instance).
But let's clarify also that locking does not lock a variable, nor does it lock a scope. It merely prevents execution of the code in the lock, if the requested lock cannot be obtained. I clarify this more in a cf summit talk I did last year, available at carehart.org/presentations). It's indeed a common misconception.
And it seems all the more important here, as using the lock as proposed here would not seem to solve the problem described. The locks would never be held longer than it takes to set or get the session var: again, neither the variable nor the scope is "locked" in the way many presume.
4) So all that said, it seems the issue is (as Zachary has put it) that an update to a session var in one request is not detected by another (for the same session) until the first finishes. And yes, it implies that Cf is not holding the session scope in memory like it normally would but instead (with Redis sessions) is relying solely on the redis store.
I find that surprising, and I'd like to prove it. Zacahry has not said what cf version he's running. It could be as old as cf2016, when Redis sessions were introduced. But the feature has received upgrades since then.
So finally, Zachary, what cf version and update level are you on, and what edition? While we're at it, what Java version is your cf using? All these are reported on the cf admin, in either the settings summary or system info pages, or within a cfdump/writedump of the server scope.
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
For sure--thanks so much for the help @Charlie Arehart and @BKBK .
I'm mostly doing testing on my local environment which uses the the developer edition. I'm also on Coldfusion 2021 update 18 and Java 17.0.7 (JDK 11.0.16.1).
To clarify, this is happening on 1 CF server instance (no cluster yet). Given the clarification of the documentation (thanks, Charlie), I think this might also be the expected behavior: the "lifecycle" of a request with this configuration (with redis sessions) would seem to be
OnRequest (get the session from redis) ->
Execute Requested Template ->
OnRequestEnd (persist session in redis).
So the modifications to a session done by a request will only be available if the session is retrieved after request 1 persists it in redis.
Copy link to clipboard
Copied
Charlie, I am at a loss what you mean when you say that locking does not lock a scope. What then are session-scoped and application-scoped locks?
It is also unclear what you mean by: "using the lock as proposed here would not seem to solve the problem described. The locks would never be held longer than it takes to set or get the session var: again, neither the variable nor the scope is "locked" in the way many presume.".
In any case, I shall explain in a moment:
- why the lock attribute scope="session" actually means that the lock's scope is session-wide;
- why session-locks are necessary when you use Redis as external session storage.
Copy link to clipboard
Copied
This is a difficult concept for me. I'm not sure I recall it exactly, but my recollection is that the CFLOCK tag in general doesn't lock the code within it. Instead, it only tells the application not to execute this code if other blocks of code with the same CFLOCK identifier are already running AND if this CFLOCK has TYPE="EXCLUSIVE". Using CFLOCK SCOPE="SESSION" won't have any effect unless there might be two requests running from the same session - multiple tabs or windows within the same browser using the same session ID, for example, or the user right-clicks on a link to open another browser tab or window. Anyway, the type doesn't mean there's automatically a lock on the session or application or whatever, unless any other code within that whatever also has the same kind of CFLOCK with the same identifier. (I used to make a lot of money based on CFLOCK bad practices, so you'd think I'd remember better, but here we are.)
Anyway, I think (1) could be true if you clearly define what you mean as the lock's scope, and you have all the code that needs to be locked within this CFLOCK or some similar CFLOCK. I'm not sure about (2) and would need to test that out to be confident about it - this doesn't mean you're wrong or anything, just that I can't independently confirm that without writing some tests.
Copy link to clipboard
Copied
@zachary_3780 , the request "lifecycle" with this configuration (involving redis sessions) is:
- OnRequestStart: ColdFusion gets the session from Redis, if the session is not yet in memory ->
- Execute requested template ->
- OnRequestEnd: ColdFusion persists the session in Redis.
The following statement is not always true: "So the modifications to a session done by a request will only be available if the session is retrieved after request 1 persists it in redis" . To see that, consider the following scenario.
The initialization session.userVar=0 occurs in onSessionStart. Suppose there are 3 simultaneously active requests from the same user. Suppose also that the following events happen in this order:
- Request 1 updates the session variable: session.userVar=5 (a write);
- Request 2 outputs session.userVar (a read);
- Request 3 updates the session variable: session.userVar=25 (a write);
- Request 1 outputs session.userVar (a read);
(Request 1 might have had a lot of other stuff to do before outputting session.userVar) - Request 2 updates the session variable: session.userVar=625 (a write);
- Request 3 outputs session.userVar (a read).
Conclusions:
- You cannot yet say which value of session.userVar has been saved in Redis. That is because you don't yet know if any of the requests has ended and, if so, which one. The first request to end will be the one whose session.userVar value is saved in Redis.
- Without the use of locks, each of the above session updates will happen instantaneously in memory. It means that, without locks, each session update will propagate instantaneously to all three requests.
As a result, the outputs will be, respectively,
in Step 2: 5
in Step 4: 25
in Step 6: 625
What this scenario demonsttrates is essentially a race condition. Various concurrent requests race for the same resource at the same time.
So, let's suppose that you want:
- requests to read only the latest update of a session variable.
- the session saved in Redis to be consistently the latest updated value.
You can achieve that as follows:
- if code reads a session variable that is updated somewhere else in the application, then enclose the code within a readonly session-scoped lock;
- if code updates a session variable that is read somewhere else in the application, then enclose the code within an exclusive session-scoped lock.
These is not merly a suggestion. You should consider it as best-practice when using ColdFusion sessions via Redis.
Copy link to clipboard
Copied
Bkbk, I meant exactly what I said. A scope lock does NOT lock a scope. It indicates the KIND of lock that you're REQUESTING, and if it can be obtained then the code in the lock will be allowed to run. (The cflock timeout limits how long it will wait to obtain that lock.)
Conversely, the code in a lock (assuming the lock is obtained) will cause that lock to be held for as long as that code takes to run. (And the lock timeout has zero impact on that duration: specifically it does NOT limit the duration.)
Dave's summary should help you also. To repeat what I'd said:
"But let's clarify also that locking does not lock a variable, nor does it lock a scope. It merely prevents execution of the code in the lock, if the requested lock cannot be obtained. I clarify this more in a cf summit talk I did last year, available at carehart.org/presentations). It's indeed a common misconception."
More specifically here is the talk and its pdf slide deck, and it included code samples and a link to still more that help solidify the point:
https://www.carehart.org/presentations/#cflock
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
Charlie, it appears to be just a difference of semantics. So I shall not squibble over it. Instead I am going to offer a demo that shows the session-lock in action.
Copy link to clipboard
Copied
It's not merely a semantic difference. I understand your stance. It's the very reason I gave the talk, to help those open to hearing.
In your test, setup one template to have a cflock on the session scope which sets a given session var and goes to sleep (in the lock) for 60 seconds, then have another template change that session var WITHOUT ANY CFLOCK.
Then in one tab start the first, then in another run the second. Does the second run? Yes. And it freely accesses and changes the session var despite the first request having a "lock".
The scope simply is NOT locked. It's a misconception, again not merely semantics. (If you mean that two requests both doing locks can block each other, which EFFECTIVELY "locks the scope", that's what it may SEEM to be doing. But again that's NOT what it's doing.)
All that said, my original reply to you on this was that the updates in Zachary's two requests will NOT block each other. They will both be able to update the session var, because the scope NOT LOCKED.
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
Charlie, look back at the discussion. You are the one who brought in the subject of "locking a scope". You attribute words to me, then proceed to point out that they are wrong. Not for the first time.
I called it a difference in semantics for a simple, practical reason. We are busy looking for the answer to a problem. The session-scoped locks that I suggest are implemented by-the-book. So long as you implement the locks as intended, what you call them is just a matter of semantics.
Copy link to clipboard
Copied
The scope simply is NOT locked.
By Charlie Arehart
No one says it is. I, in particular, have been talking all along about "session-scoped locks". In any case, I have included code, where necessary, to clarify what I mean. The proof of the pudding is in the eating.
All that said, my original reply to you on this was that the updates in Zachary's two requests will NOT block each other.
By Charlie Arehart
It is unnecessary to tell me that. It is clear from the discussion that I am aware of it. Otherwise, why else would I be suggesting. as an alternative. a lock to enable requests to "block" each other?
Copy link to clipboard
Copied
I was referring specifically to if they ARE locked as you propose. And as such, I still stand by all I said, including why I brought it up ("The locks would never be held longer than it takes to set or get the session var: again, neither the variable nor the scope is "locked" in the way many presume.) as well as the rest I offered on locking.
You can remain in your stance. I'll let readers decide between our positions objectively.
Most important, Zacahary can tell us if your proposed solution helps with his problem. I appreciate that's as much your goal as mine (and Dave's).
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
Session-scoped locks and session variables - a demo and proof of concept
- Save the following 4 CFM files in a directory under the web root.
- Then run them in that order, in quick succession.
sessionTest1.cfm
Session.myVar initialized to 0, then updated to Session.myVar + 1.<br>
<cfset session.myVar = 0>
<cfset session.myVar = session.myVar + 1>
<cfoutput>
Current value of session.myVar is: #session.myVar# <br>
</cfoutput>
sessionTest2.cfm
<strong>
Case 1: No session lock.<br>
</strong>
<cfset startTime = getTickCount()>
<cfset session.myVar = session.myVar + 1>
Session.myVar updated to session.myVar + 1. <br><br>
<cfoutput>
Current value of session.myVar is: #session.myVar# <br>
Elasped time in milliseconds: #getTickCount() - startTime#
</cfoutput>
sessionTest3.cfm
Session.myVar updated after 30 seconds to session.myVar + 1 within exclusive session-scoped lock.
<cflock scope="session" timeout="30" type="exclusive">
<cfset session.myVar = getUpdatedSessionVar()>
</cflock>
<cffunction name="getUpdatedSessionVar" returntype="numeric">
<!--- Wait 30000 milliseconds, that is, 30 seconds --->
<cfset sleep(30000)>
<!--- Update session.myVar --->
<cfset session.myVar = session.myVar + 1>
<cfreturn session.myVar>
</cffunction>
sessionTest4.cfm
<strong>
Case 2: Read session.myVar with readOnly session-scoped lock.<br>
</strong>
<cfset startTime = getTickCount()>
<cflock scope="session" timeout="35" type="readonly">
<cfoutput>
Current value of session.myVar is: #session.myVar# <br>
Elasped time in milliseconds: #getTickCount() - startTime#
</cfoutput>
</cflock>
Demo results:
- sessionTest2.cfm updates the session variable set by sessionTest1.cfm and outputs the updated value instantaneously, without any time lapse.
- sessionTest4.cfm waits for the update in sessionTest3.cfm to complete before outputting the updated session variable.
Copy link to clipboard
Copied
For any readers seeing this thread, note that on a mobile phone you may not be able to tell that this previous reply from bkbk is from well before the previous few messages above it.
He wrote this one (with his sample code) several minute after his initial "semantics" contention. But I had replied to that above BEFORE he wrote this, and then overnight he's responded more than once to that, though theses are appearing well ABOVE this.
In other words, this post with his lock code example is not the current "end of the discussion" started by his "semantics" comment, though a casual reader might conclude it was. The messages preceding it were written AFTER it was. It's unfortunate this can sometimes happen and be hard to discern on a phone.
/Charlie (troubleshooter, carehart. org)
Copy link to clipboard
Copied
@zachary_3780 , to cut a long story short, my suggestion in brief:
- Suppose a user of your application may generate multiple simultaneous requests, each of which can update a session variable., session.userVar. Then that is potentially a race condition.
So, you should place, within a session-scoped lock of "exclusive" type, any code that updates session.userVar . You should also place, within a session-scoped lock of "readonly" type, any code that reads session.userVar . This is best-practice in software development when dealing with potential race conditions. - Suppose that you still face the issue after applying locks. Then it is very likely that we're dealing with a bug. In that case, alert the forum and. even better, report a bug.
Copy link to clipboard
Copied
Hello friends--I have also provided my own reproduction:
application.cfc
public void function onRequestEnd(TargetPage) {
if (TargetPage == '/test2.cfc' || TargetPage == '/test3.cfc') {
session[TargetPage & ' end time'] = now()
}
return
}
public boolean function onRequestStart(required string TargetPage) {
if (TargetPage == '/test2.cfc' || TargetPage == '/test3.cfc') {
session[TargetPage & ' start time'] = now()
}
return
}
test.cfm
<cfif !SESSION.keyExists('myVariable')>
<cfset SESSION.myVariable = 0>
</cfif>
<!DOCTYPE html>
<html>
<head>
<title>Test Page</title>
<script>
async function sendRequests() {
fetch('test2.cfc?method=request1')
.then(function(response) { return response.text(); })
.then(function(data) { console.log('Response from request1:', data); })
await setTimeout(250)
fetch('test2.cfc?method=request2')
.then(function(response) { return response.text(); })
.then(function(data) { console.log('Response from request2:', data); })
}
</script>
</head>
<body>
<button type="button" onclick="sendRequests()">Click Me</button>
<cfdump var=#SESSION#>
</body>
</html>
test2.cfc
component {
remote numeric function request1() returnformat="JSON" {
sleep(2000);
lock scope="session" type="exclusive" timeout="3" {
session.myVariable = 2;
return session.myVariable;
}
}
remote numeric function request2() returnformat="JSON" {
sleep(2000);
lock scope="session" type="readonly" timeout="3" {
return session.myVariable;
}
}
}
In sum, myVariable is still 0, even though the request1 is return the new value (2), but request 2 overwrites not only myVariable, but also the /test1.cfc start time and /test1.cfc end time variables. Basically, the whole session is persisted with no care for what was already in the session.
Correct me if my implementation was poor, but it seems the locks are not solving this issue.


-
- 1
- 2