Serialization Of Structs

[craigomgwtf]

Our content management system uses a very complex database structure to represent object data. In order to improve website performance, we publish the object data to the web servers as binary objects. They are serialized FastHashtable objects - basically structs of structs and arrays. (Not CFC's)

We do it like this:

<cffunction name="ObjectToBinary">
<cfargument name="theObj">
<cfscript>
var bs = CreateObject("java","java.io.ByteArrayOutputStream").init();
var out = CreateObject("java","java.io.ObjectOutputStream").init(bs);
out.writeObject(theObj);
out.close();
return bs.toByteArray();
</cfscript>
</cffunction>

<cffunction name="BinaryToObject">
<cfargument name="bytes">
<cfscript>
var bs = CreateObject("java","java.io.ByteArrayInputStream").init(bytes);
var inStream = CreateObject("java","java.io.ObjectInputStream").init(bs);
var obj = inStream.readObject();
inStream.close();
return obj;
</cfscript>
</cffunction>

This is working great for our CF7 server instances. The problem is that we can't use these objects with CF8 because the FastHashtable object has changed in the CF API. I had an idea to use serialized java.util.HashMap objects instead of FastHashtable objects because, as they impleiment the Map interface, they should be able to cast into a FastHashtable, but CF8 is still complaining about binary versions.

Does anyone have any ideas of how to use the same binary struct compatible objects in both CF7 and CF8 (and moving forward with future JVMs?) BTW WDDX is not a good option for us because of the large size and decoding time.

[BKBK]

Your code is basically Java, so there's nothing wrong with it. At least, not in Coldfusion terms. You cannot cast with java.util.HashMap because it is not in FastHashtable's hierarchy. Run the following

<cfset x=createobject("java","coldfusion.util.FastHashtable")>
<cfset sup = x.getClass().getSuperClass().getName()>
<cfset sup_sup = sup.getClass().getSuperClass().getName()>
<p>sup: <cfoutput>#sup#</cfoutput><br>
sup_sup: <cfoutput>#sup_sup#</cfoutput></p>

You will see that the hierarchy is
coldfusion.util.FastHashtable => coldfusion.util.CaseInsensitiveMap => java.lang.Object

[craigomgwtf]

That may be true, but as you know from the ColdFusion docs, you can cast any Java Map into a coldfusion runtime struct:

<!--- origional object --->
<cfset object0 = structnew()>
<cfset object0.foo = "bar">
<!--- serialize as hashtable --->
<cfset htObject = CreateObject("java","java.util.Hashtable").init(object0)>
<cfset binary = objectToBinary(htObject)>
<cfset object1 = structnew()>
<cfset object1.putAll(binaryToObject(binary))>
<cfoutput>
<h3>Serialized Hashtable</h3>
origional object class = #object0.getClass().getName()#
<br>cast object class = #htObject.getClass().getName()#
<br>deserialized object class = #object1.getClass().getName()#
<br>string lengths = #object0.toString().length()# : #object1.toString().length()#
<br>decode serialized hashtable success? #object0.equals(object1)#
</cfoutput>

The PROBLEM is that even when I save a java.util.Hashtable in CF7 and try to read it back in CF8 i get the following error:

coldfusion.util.FastHashtable; local class incompatible: stream classdesc serialVersionUID = -7779663176687694609, local class serialVersionUID = -2236937973785605209

I am confused about why the error comes from coldfusion.util.FastHashtable instead of the java.util.Hashtable, or why it blows up at all... My goal is to save a runtime struct in the way that it is fastest to retrieve and use again (in a persistent data store).

[-__cfSearching__-]

craigomgwtf wrote:
> That may be true, but as you know from the ColdFusion docs, you can cast any Java
> Map into a coldfusion runtime struct:

Conversion is not the same as serialization. From what little I have read, it does not make any guarantees about serialization.


> <cfset object0 = structnew()>
> ...
> I am confused about why the error comes from coldfusion.util.FastHashtable instead
> of the java.util.Hashtable, or why it blows up at all...

Because that is the class of the object you serialized (ie "object0" is an instance of coldfusion.util.FastHashtable). Based on the error, CF8 is trying to deserialize it into its version of FastHashtable, but cannot due to a serialVersionUID difference.

I know very little about serialization beyond that fact that serialVersionUID is used to determine compatibility. In theory serialVersionUID should be incremented when a class changes significantly or in a way that is not backward compatible. In other words, it can no longer reliably handle objects serialized with a different version . That is what the error message is saying: it cannot deserialize the object because its version UID is incompatible with CF8's version UID.

[BKBK]

I copied the following code from you, word for word. It runs without error, and displays:

Serialized Hashtable

origional object class = coldfusion.runtime.Struct
cast object class = java.util.Hashtable
deserialized object class = coldfusion.runtime.Struct
string lengths = 11 : 11
decode serialized hashtable success? YES

I am on Coldfusion 8 Updater 1, that is, CF8.0.1.191463.

[-__cfSearching__-]

> I copied the following code from you, word for word. It runs without error

Yes, unfortunately it does not work if you serialize the structure using MX 7,0,2,142559 and attempt to deserialize it with CF 8,0,1,195765. Because the classes have different serialVersionUID's, the deserialization fails.

[BKBK]

-==cfSearching==- wrote:
... it does not work if you serialize the structure using MX 7,0,2,142559 and attempt to deserialize it with CF 8,0,1,195765. Because the classes have different serialVersionUID's, the deserialization fails.

Thanks, -==cfSearching==- . That's indeed my point. To avoid problems, you should serialize and deserialize within the same version. Very much like when you compile in one step and run in the next. There is bound to be an error when you switch environments, for example, when you switch JVM versions.

[-__cfSearching__-]

> To avoid problems, you should serialize and deserialize within the same version.

Very true. Unfortunately, it sounds as if they are serializing with MX7, storing the data, then attempting to deserialize it back into CF8. Normally, I would suggest both environments use a shared jar. But as it involves core CF classes, that strikes me as a very bad idea. Assuming it would even work in the first place ..

[BKBK]

> it sounds as if they are serializing with MX7, storing the data,
> then attempting to deserialize it back into CF8.

That's what I assume, too. The serialization is done in JVM v1.4.x and the deserialization in JVM v1.6.x. The serialVersionUID error is then inevitable. It's a problem in Java to serialize and deserialize between different JVM versions.

However, Craigomgwtf could borrow a trick from Java. One workaround in Java is to modify the class to be serialized by hard-coding one extra line in it.

static final long serialVersionUID = -2236937973785605209L;

This line simply defines a class constant. The JVM will be looking for it.

This is handy because it means we could make use of Coldfusion's flexible runtime capabilities. We could just include the constant later, on the fly. For example, Craigomgwtf could serialize as follows

<cffunction name="ObjectToBinary">
<cfargument name="theObj">
<cfset serialVersionUID = -2236937973785605209>
<cfset arguments.theObj.serialVersionUID = JavaCast("long", serialVersionUID)>
<cfscript>
var bs = CreateObject("java","java.io.ByteArrayOutputStream").init();
var out = CreateObject("java","java.io.ObjectOutputStream").init(bs);
out.writeObject(arguments.theObj);
out.close();
return bs.toByteArray();
</cfscript>
</cffunction>

Just a whacky thought.

[BKBK]

Craigomgwtf,

Here's a proof of concept. I serialized in MX7 with serializeObject.cfm and deserialized in CF8 with deSerializeObject.cfm without any problems. I used the MySQL5 database, and had to include the mysql-connector-java-5.0.8-bin.jar in MX7.

[-__cfSearching__-]

> Here's a proof of concept.

It definitely works, but there are a few "gotchas". Because the object would be deserialized back to a Hashtable, it would be case sensitive, unlike a ColdFusion structure. So key searches (structKeyExists, etcetera ..) would be less tolerant. Plus, if the object contains nested structures all of them must be converted to Hashtables, not just the top level. Too bad we do not have the CF API. I imagine there are similar issues deserializing other complex objects...

> One workaround in Java is to modify the class to be serialized by hard-coding one extra line in it.

Yes, I was thinking about that too but changed my mind. I am always hesitant to mess around with core CF classes. Without an API, it is difficult to predict the affect of changes. Plus, my assumption would be that the version numbers are changed deliberately. To warn of potential compatibility problems. While I know that might not be the case, overriding it without knowing one way or the other just felt risky to me.

[BKBK]

-==cfSearching==- wrote:
Yes, I was thinking about that too but changed my mind. I am always hesitant to mess around with core CF classes. Without an API, it is difficult to predict the affect of changes. Plus, my assumption would be that the version numbers are changed deliberately. To warn of potential compatibility problems. While I know that might not be the case, overriding it without knowing one way or the other just felt risky to me.

You are right. Suppose the workaround cuts the mustard now, what happens then when a new JVM version comes out? The last thing you want in your design is to have to keep changing the plumbing whenever a new tenant moves in.

I agree with you. It is all very desperate. I am just trying to help Craigomgwtf through the night. It aint funny to have a bunch of MX7 objects that you cannot use in CF8. One must accept the risk that a lot of previous work might go to waste.