I'm hoping someone could shed some light.
We're testing Redis as session storage. It works just fine, but we're running into an issue where the keys are not properly expired by Redis - they basically just fill up the store. I think the problem is that the keys don't have an expiration?
Greatly appreciate any insights.
We're on CF2016, SP7.
I am not aware of there being a generic problem of that sort, but you say you are on CF2016 "sp7". By that I assume you mean "update 7", and to be clear there are 6 more updates you have not applied. Maybe if you do those updates, something among their hundreds of bug fixes may well address this issue. You could dig through the technotes for each update, and their link to the page for "bugs fixed" in each update. Or just do the latest update, and try again.
I will note that even if things were "fixed", then by an update, I would NOT expect it to affect the EXISTING sessions in redis. But it should (if there was a fix) affect sessions created from that point forward.
Let us know if you give that a try. Otherwise, you would want to look into redis to see what info it has for the expiration of the keys that CF is creating. There are free tools like redisinsight that can help with that.
Thanks, Charlie --
Yes. I've planned on applying those updates as well. Devs are in the process of testing those updates before we apply them to our test environment.
Additional info - example:
127.0.0.1:6379> TTL cf:sessions:metadata:5809_4ca8573ff508a83d-7F867DF7-076A-99F3-A7C10E2598299662
According to Redis docs:
The command returns -1 if the key exists but has no associated expire.
In fact, this environment I'm testing is local and it has CF2016 Update 13 applied. So the updates isn't the fix.
are you being sure to test the sessions that are being created most recently? you can see the sessionid with a dump of the session scope.
Yes sir. These are brand new sessions.
OK, got me. I have to admit I have not bothered to check closely to see if that was the case (that the keys CF created for its sessions had an expiration). We certainly would expect them to...though I will say that as long as CF internally somehow keeps its own sessiontimeout for the session, then as long as CF did expire the sessions, that may be why no one has noticed.
But certainly if they were just allowed to pile up in redis, unchecked, that would be bad.
If you do get any better monitoring on Redis, you may find more about the comms between CF and Redis.
But assuming things are as you say (that CF is not setting an expiration), and if you are somehow able to tell that these go back a long time, and have CF session expirations that have long-since expired, then you have a good question for Adobe.
We can hope that someone may chime in here, but there's no certainty. Sometimes they do, most often they do not. You may want to instead open a bug report, at tracker.adobe.com. I will say I just did a search there, for any for CF with redis in the title, and of the 12 that came up, none seemed to be related to this issue (judging by the title shown), so you may well have a new issue that deserves noting. And maybe someone else will chime in if they've seen it but had not bothered to create a ticket.
Do be sure to indicate that you're on CF2016 update 13. And might you be in a position to test things with CF2018 update 7? That could be helpful to report. (I could, but I am going from call to call so only have time to offer this reply, not setup such a test, for now.)
Hi, Charlie -
I'm not in a position to test CF2018 as we don't have any environment running in 2018 just yet. That's not to say I can't install and test 2018, but at the moment, we are still in CF2016 version.
It's possible that Adobe had addressed the issue in the later patches (Update 8+) - but I went through all the documents and bugs that were fixed on all the updates, but it didn't mention anything about Redis - it's possible that it's marked differently in the bugs that were fixed.
What is the value of the sessionTimeout setting in your Application.cfc/Apllication.cfm file? If it is a large number, then ColdFusion might not expire sessions.
sessionTimeout is set to 30 minutes.
Jojo, I have done some more investigating, and I think I have what should be seen as good news. The bad news is that I'm not sure things are quite how you have described them to be.
Let me be clear: I'm not denying there's no TTL on the keys. But I am saying that I've found that the sessions do not pile up in redis, unchecked. CF is itself managing the deletion of the keys, when IT detects that the session times out (per the timeout set for the session, in CFML). I have confirmed this with both CF2018 and 2016, with both their latest updates and earlier ones.
Let me be more specific: you opened the thread here saying that redis session keys created by CF's redis storage feature (new in CF2016) are "not expiring", and that "they basically just fill up the store", which you affirm because the redis CLI's TTL command shows that "the key exists but has no associated expire".
While it is true that the keys CF creates have no ttl, it is NOT true that the keys "never expire" and therefore "fill up the store". First, I could see in Redis (via its cli, or other tools) that a session I created to last for, say, a minute did in fact disappear from both CF and redis after a minute (assuming there was no interaction with CF for that session id, for that minute). I could even see it in as short as a 10-second cf session timeout (I did not test still shorter timeouts).
Second, besides using the CLI or other tools (such as redisinsight or redis commander) to show whether the keys existed in redis or not, I also ran a profiler against redis (the one included with the free redisinsight tool), and I watched as the timeout passed, and I did see CF issue to redis the hdel and srem commands to remove the keys (and sets and hashes) associated with a given CF session.
So I will ask you: are you saying that you did NOT see such deletion of the keys/sets that CF is creating for a given session? (For those not familiar with redis, a set is a group of related keys, and the session var names and values for a given session are stored within such a "set" in redis.) Each session also has some related "timeout" and "metadata" information.
Perhaps you ARE finding some session data remains inside redis, but an interesting question is whether a) perhaps a problem can happen where CF doesn't delete sessions if CF crashes during the timeout, or b) perhaps CF uses that timeout or other data tracked (one of which is a "max inactive interval", which reflects the internal CF session timeout interval) to run clean up tasks even after such a crash. I don't know, and some of the keys and their values are encoded and hard to parse, but maybe with this additional info and encouragement you or someone else may want to dig in further to see what's going on in that respect.
But I can say with 100% confidence that CF IS indeed deleting sessions from redis when they timeout within CF. I'd be curious to hear more about what you are seeing that seems to be "sticking around longer". Maybe if you or someone else figures out the encoding of the data within the keys and related metadata/timeout info, you might find that what you have are sessions with just really long session timeouts. (The default setting in the CF Admin is that the max session timeout is 2 days, while the default session timeout is 20 mins. Those could be changed in the CF Admin, or the timeout can be set in your app, of course.)
And of course, as long as a request is made on behalf of a session, that session will not timeout--so sessions can indeed last much longer than "the session timeout". (I added this sentence after my initial posting of this reply.)
Maybe someone from Adobe will chime in with still more details to help about the above. But I would love to hear what you think of what I've shared, and maybe you might share if you are using anything other than the CLI to see what's in there.
One last thing: it can of course be challenging trying to watch what's going on against a redis that's being used with CF sessions in production, in that there will be so much data there. If you could setup a simple independent redis instance (like I did), you may see things more readily like I did. One can do that using any of many means, from available installers (that could be implemented locally for free, along with a free CF dev edition), or using Docker (for CF or Redis or both), or using available redis services (including redis labs, which offers free and commercial cloud deployments, so you need install nothing).
Hope that's helpful.
Great investigation, Charlie!
Contains much more information that will be handy for anyone using Redis in ColdFusion.
Thanks, Charlie. Yeah, I had the assumption that CF is managing those sessions given that the keys don't have an expiration. With that said, I know for a fact that our test environment, which is running under Update 7, is not deleting those keys - I have flushed all the keys and started fresh and I could see the storage gets full - but for whatever reason, CF is not removing or unable to remove the keys.
With that said, I cannot replicate the same scenario in my local environment - which is running under Update 13. It's possible that the latest updates fixed whatever the issue is. I did look through the update release notes and bugs fixed, but no mention of Redis - that's not to say that it wasn't addressed.
One thing I did notice was the exponential growth of the DB size when scheduled tasks are running. We have about a dozen scheduled tasks that run every few seconds and that seems to generate a lot of data into the memory. I'm not entirely sure if it's generating a session object each time the job runs or not. But as soon as I paused all the scheduled jobs, the exponential growth stopped and the DB size started to normalize to the actual human session objects.
I'll find some time this weekend to install those updates to our test environment and retry the test again. I will post back.
Thanks as well for those tools that you mentioned.
jojoserquina , a test you could do when the scheduled tasks are on, and then again after you switch them off (in a test environment, NOT in a production environment):
1) Restart ColdFusion, ensuring there are no current sessions
2) Clear the Redis table of all stored sessions
3) Run the scheduled tasks.
on a ColdFusion page, dump all current sessions
in Redis, dump all currently stored session keys
127.0.0.1:6379> keys *cf:sessions*
Record these results.
4) Stop de scheduled tasks
on a ColdFusion page, dump all current sessions
in Redis, dump all currently stored session keys
127.0.0.1:6379> keys *cf:sessions*
Compare the results, especially number of session keys
jojo, to your observation about sessions: yes, each execution of a scheduled task will create a new session, assumign that the page being called is in a directory that has an application.cfc or cfm in it (or in a directory above it). Just like any CF page, such a CF page would execute that application file, and if sessionmanagement is enabled, then it would support session vars.
And whenever a CF page is called from ANY sort of automated mechanism (as scheduled task, via cfhttp, from a web service client, from a spider, from a load test tool, from a security scanning tool), that page will call into CF without any cookies (typically), and so CF will create a NEW session for each such call.
So yes, this can put pressure on creation of sessions in Redis (just like it would put pressure on memory in CF, regarding the CF sessions themselves).
But none of it should explain why CF would not REMOVE the sessions, as I observed. Also, I said I had observed that removal in both the latest updates to CF2018 and 2016 as well as in older ones, including even update 6 (from Apr 2018). So no, I don't think it's "new behavior" that it does that.
Finally, I will say that you (or anyone) can easily test all this without need to "install" CF (or "risk applying updates") by using the Adobe CF Docker images. That's what I was doing to demo this behavior--and the oldest Docker image they offer is indeed for that update 6. I realize you may not be familiar with Docker and so may not see this as an "easy way" to try different versions. I have more introducing it here:
Let us know how things go, however you proceed.
Thanks, Charlie -- it is indeed the scheduled tasks that were straining the Redis server.
I'm trying to get a docker environment going - just running into some minor issues, which I've posted in the slack community. Some had offered alternate solutions - it's around the env file execution.
Anyway, yeah we're done with this post. Thanks!
Thanks for your update on things, JoJo.
As for your challenge with Docker (by which I assume you mean a challenge with the Adobe CF docker images), you may want to ask in a forum thread here. There are many folks (myself included) who while VERY active and responsive here may not look at Slack at all, or rarely. You may be missing out on their assistance.
Indeed, there may not be many people who have worked with the Adobe CF Docker images as much as I have, as I have been presenting on them and writing about them for over a year. (Of course, there are many with still more years experience working with the Commandbox CF Docker images provided by Ortus. They have different environment vars, and that's why I am suspecting your problem may be unique to the Adobe-provided images. They are case-sensitive, btw, in name and value.)
But again, please don't ask the question in reply here. It really deserves its own forum thread.
I am finding a similar issue. We are running CF2018 Update 9 (latest as of this writing) with session storage in Redis and finding that sessions essentially are never expiring. In CFAdmin we have configured 30 minutes on maximum session timeout, 30 minutes on default and in Application.cfc a 30 minute timeout configured for session.
EDIT: I have posted this into a new thread since this one is 'done' and it sounds like this may be a different issue.