Are There Limits to What CF Can Store in Redis?

[sdsinc_pmascari]

For all our caching needs we've always used the built-in EHCache in ColdFusion.  It's served us well.

As our caching use has grown, lately, we're looking to offload our cache duties to an external service.

To be clear, in this context I'm not talking about session cache but, rather, caching of specific data using ColdFusion's built-in caching functions.

Our apps have several long-running processes that build reports for end users.  As the processes run, we cache the data so as to speed up any subsequent calls.  The data cached can be rather extensive.  Large structs holding any number of additional structs, arrays, queries, etc...

Using EHCache we've never had issues.

However, as we're trying our Redis, using a free account at redis.io, we're finding the Redis cache doesn't seem to like our large data objects.

When I dump out the data obtained from cache via cacheGet() function, instead of seeing a nice ColdFusion structure, we get what appears to be a JSON string but it doesn't validate as JSON.  In addition, there look to be internal CF function calls in the string that would be assembling the data such as queries and such.  For instance, here's an excerpt from one of the JSON strings:

TRADENEWS={{trades={[Table (rows 20 columns engagements, Headline, datepub, hreflink, author, image, source): [engagements: coldfusion.sql.QueryColumn@61138795] [Headline: coldfusion.sql.QueryColumn@52942e28] [datepub: coldfusion.sql.QueryColumn@2dd401bd] [hreflink: coldfusion.sql.QueryColumn@5a8d982] [author: coldfusion.sql.QueryColumn@787ad98d] [image: coldfusion.sql.QueryColumn@bec4dc4] [source: coldfusion.sql.QueryColumn@6b86ced] ]},ERROR={NONE}}},MILERADIUSLIST={216},AUDIENCEINFO={{AudienceReport={[Table (rows 1 columns ID, AIAGROUPID, AIACATID, CATEGORYNAME, GENERALDESC, OTHERINFO, WIKIPEDIAURL, WEBSITEURL, WEBSITENAME, WEBSITEURL2, WEBSITENAME2, SELECTEDLOCALID, SECONDARYLOCALID, OTHERTOPPROSPECTS, CEXCODE, SECONDARYCEXCODES, PEAKMONTHS, PEAKMONTHSDESC, LOCALNEWSSOURCE, AUDIENCEIMAGE, SELECTEDHEALTHCARE, IAB, COMPOSITION, OMITCOMPOSITION, PRIMARYPURCHASEINTENT, SUPPRESSGENDERAGE): [ID: coldfusion.sql.QueryColumn@459fc171] [AIAGROUPID: coldfusion.sql.QueryColumn@6b07cefe] [AIACATID: coldfusion.sql.QueryColumn@26c2c37] [CATEGORYNAME: coldfusion.sql.QueryColumn@4776e793] [GENERALDESC: coldfusion.sql.QueryColumn@7c593245] [OTHERINFO: coldfusion.sql.QueryColumn@297c630e] [WIKIPEDIAURL: coldfusion.sql.QueryColumn@4a92ef82] [WEBSITEURL: coldfusion.sql.QueryColumn@3a28d33c] [WEBSITENAME: coldfusion.sql.QueryColumn@fd61526] [WEBSITEURL2: coldfusion.sql.QueryColumn@45f873c1] [WEBSITENAME2: coldfusion.sql.QueryColumn@18cb7a03] [SELECTEDLOCALID: coldfusion.sql.QueryColumn@6486fb15] [SECONDARYLOCALID: coldfusion.sql.QueryColumn@5bc6bd86] [OTHERTOPPROSPECTS: coldfusion.sql.QueryColumn@5da6bb44] [CEXCODE: coldfusion.sql.QueryColumn@1efed991] [SECONDARYCEXCODES: coldfusion.sql.QueryColumn@4d73c708] [PEAKMONTHS: coldfusion.sql.QueryColumn@23b769ad] [PEAKMONTHSDESC: coldfusion.sql.QueryColumn@4b239a04] [LOCALNEWSSOURCE: coldfusion.sql.QueryColumn@2379f0a0] [AUDIENCEIMAGE: coldfusion.sql.QueryColumn@70e13840] [SELECTEDHEALTHCARE: coldfusion.sql.QueryColumn@6803ddf0] [IAB: coldfusion.sql.QueryColumn@3fa7894e] [COMPOSITION: coldfusion.sql.QueryColumn@53b792ea] [OMITCOMPOSITION: coldfusion.sql.QueryColumn@2a6d83e4] [PRIMARYPURCHASEINTENT: coldfusion.sql.QueryColumn@5c772704] [SUPPRESSGENDERAGE: coldfusion.sql.QueryColumn@5f19383d] ]}

Whereas, if we dumped the data out directly, with no caching, or from EHCache, we'd get a typical CF object such as:

Is there a setting for Redis or CF to enable the caching of all CF data types and/or larger objects?

[Charlie Arehart]

Paul, while you may await Adobe or others, I'll say first I'm not aware of any such setting related to what is cached based on size or type.

As you've surely noticed, there IS the option on the cf admin caching page to control WHICH caching mechanism to use by default, which can be overidden at the app level one. And it's only in cf enterprise (or the free dev or trial edition) that we can use an external cache engine/service like redis or memcached, while CF standard only offers ehcache and jcs (as does ent).

So about your issue, first are you confirming that if you use cacheGet() against a query cached in ehcache, it shows none of that junk? 

And are you saying that if it's a small query, things look ok when cached in redis? It's just not clear. 

Finally, since you're using the free edition of redis cloud, it does have limits. You could be hitting that. If you restart it (the redis service), does a small query work? Then does a larger one work (perhaps only for a while)? 

Note that you can monitor what's going on within redis using either the redis-cli or tools like redisinsight (as well as redis cloud's own ui). These also let you watch all the communication to and from redis, which might offer useful info. 

Let us know how it goes. 

[sdsinc_pmascari]

there IS the option on the cf admin caching page to control WHICH caching mechanism to use

This is what we are testing.  We are running Enterprise and have changed the CFAdmin setting to Redis with settings from redis.io

are you confirming that if you use cacheGet() against a query cached in ehcache, it shows none of that junk?

Yes.  I have never seen CF dump data like this when using EHCache.  I don't know if it's only choking on queries or what.  The object we're storing is big.  Full of structs, arrays, queries, etc....

And are you saying that if it's a small query, things look ok when cached in redis?

Yes.  Just as a test, I put together a quick test and it works as expected in Redis.  (NOTE: This is substantially smaller than the data structures we normally cache)

<cfset cacheKey = "ThisIsATest123">

<cfif isNull(cacheGet( cacheKey ))>
	<cfset fromCache = false>
	<cfset testData = {
		WasThisATest = true,
		HaveABeer = true
	}>
	<cfset testQry = queryNew("id")>
	<cfloop from="1" to="10" index="i">
		<cfset queryAddRow(testQry,{id=i})>
	</cfloop>
	<cfset testData.testQry = testQry>

	<cfset cachePut(cacheKey, testData , createTimeSpan(1,0,0,0),createTimeSpan(0,1,0,0) )> 
<cfelse>
	<cfset fromCache = true>
	<cfset testData = cacheGet( cacheKey )>
</cfif>
<cfdump var="#fromCache#">
<cfdump var="#testData#"><cfabort>

the free edition of redis cloud, it does have limits

I updated our test connection in redis.io for more space and connections.  No effect.  Still dumps out junk.

you can monitor what's going on within redis using either the redis-cli or tools like redisinsight

Yep, I have been looking through the redisinsight app.  The value it shows in the cache is the same garbage:

[Charlie Arehart]

Ok on all that. And perhaps others (especially Adobe) may confirm knowing or recreating the issue. (I'm generally on mobile as I have time to check into the forums here.)

In the meantime, some more thoughts:

A) Are you saying that if you run the code that fails with Redis but switch it to use ehcache, it WORKS for that exact SAME data? Again, you could use the app-level settings to change it, if you're reluctant to change it in the admin for any reason.

B) As for redis, are you confirming that you're really creating new data (since making the change in redis), as opposed to perhaps seeing data created previously?

C) If yes, to both the above, what if you do a cfdump of the query right when you store the data in redis? Even if you "can't", because of ui issues, as you may know you can direct cfdump to write to a file (whether as html or as plain text). I just want to confirm that the "junk" is not somehow in the query BEFORE you pass it to redis to be stored.

I appreciate that you may have either done all these or you may find they proven fruitless (though please consider trying them if you've not). Even if this gets us nowhere, again it was something to consider while awaiting others. And/or it may help save them considering or proposing the same. 🙂 

[Charlie Arehart]

Paul replied below saying yes to my first two points, a and b. But Paul, can you please consider then my point c here? 

[BKBK]

@sdsinc_pmascari ,  Apparently, the result displays to the client the underlying ID of the Java object representing each column.  That suggests a problem in the integration between the ColdFusion and Redis engines. Therefore, that seems to me to be an integration error. You should report a bug.

[Charlie Arehart]

FWIW, note that it's not ALL queries that exhibit the problem, but only some (or under some conditions). While it seems clearly a bug, it would seems helpful for him/any of us to try to discern a bit more, such as to make a replicable demo for such a bug report. Otherwise Adobe may be unable to understand, let alone resolve it. 

[BKBK]

As far as I can see, @sdsinc_pmascari not only has a replicable demo, he has published it here.

I trust he will attach his code and the result to the bug report.

[sdsinc_pmascari]

Charlie, to answer your 2 questions: Yes and Yes.  The code fails in Redis but when I switch to EHCache it works, and yes, I've verified it's new data.

I've reached out to CF Support and they've asked for a script to reproduce.  I'll have to figure out how to give them something, since the code I'm using is part of our apps and has a myriad of different data sources, objects, etc feeding it.

I'll report back what I find.

[Charlie Arehart]

Paul, I'm going to reply to this in the thread where I wrote what you're replying to.  It will make more sense both for those who may read that thread and  because I'll refer to something else I'd said there. (Sadly, this forum doesn't let even us moderators move comments from one thread to another. ) 

[Charlie Arehart]

BKBK, yes, his code demos what he said DOES work for him, even with Redis. The point is that we need to somehow find when it fails. That's my understanding, at least. 

[sdsinc_pmascari]

OK, this took quite a bit of time, but we've narrow it down to rare instances when a function includes its own arguments as part of its return payload.   The Redis cache apparently doesn't handle this?

Take this code example we made up and just sent to Adobe:

<cfset cacheKey = "ThisIsATest123">

<!--- <cfset cacheremove(cacheKey)> --->

<cfif isNull(cacheGet( cacheKey ))>

	<cfset fromCache = false>

	<cfset testData = {}>


		<cfset testData = WTFFxn("test1","test2")>

	<cfset cachePut(cacheKey, testData , createTimeSpan(1,0,0,0),createTimeSpan(0,1,0,0) )> 

<cfelse>

	<!--- <cfset cacheremove(cacheKey)> --->
	<cfset fromCache = true>
	<cfset testData = cacheGet( cacheKey )>

</cfif>

<cfdump var="#fromCache#">
<cfdump var="#testData#"><cfabort>
<!--- --------------------------------------------------------------- --->
<cffunction name="WTFFxn">
	<cfargument name="wtf1" required="true">
	<cfargument name="wtf2" required="true">

	<!--- If returned like this: Redis ERROR --->
	<cfset ret = arguments>

	<!--- If returned like this: Redis ERROR --->
	<cfset ret.wtf1 = arguments.wtf1 >
	<cfset ret.wtf2 = arguments.wtf2 >

	<!--- If returned like this: Redis OK --->
	<cfset ret = {
		wtf1 = arguments.wtf1,
		wtf2 = arguments.wtf2
	}>

	<cfreturn ret>

</cffunction>

We're calling the 'WTFFxn" function which simply returns the arguments we send in.  We tried 3 methods of returning:

• Simply sending the arguments right back out, ret = arguments, results in the Redis 'error'
• Assigning the arguments to a new structure using dot notation, results in a Redis 'error'
• However, if we reassign the arguments to a new structure using curly braces, this is handled properly by Redis cache

Bizarre, and we normally don't return arguments in this way, but we found this code leftover from a dev's troubleshooting a completely different issue!

Again, this only happened to us when using Redis caching.  EHCache handles it no problem.

Anyway, seems like a bug.  I've reported to Adobe support and will also submit it as a bug.

Are others able to reproduce this?