2016 to 2018 migration with clustered scheduled tasks?

[yuliangr39032863]

has anyone done this?  We just upgraded one of our nodes and CFIDE is giving us an db incompatibility error.

Windows 2016 Server

MSSQL Server 2017 cu13

CF2016 Enterprise

CF2018 Enterprise

the clustered scheduled tasks was initialized with CF2016

we try to go to CF2018's CFIDE Scheduled Tasks and attach this new node to the existing datasource for the cluster that has existing running schema.

is it possible CF2018's clustered scheduler is not compatible with CF2016's clustered scheduler?

[yuliangr39032863]

just compared the qrtz_* table schemas between 2016's and 2018's.   they're identical..

[yuliangr39032863]

i did a tiny bit of CFIDE hacking.

edited \cfusion\wwwroot\CFIDE\administrator\scheduler\scheduleTasks.cfm

line 158 in the try catch block to push the exception message to efrErrorMessage

I get the raw error on the screen :

Couldn't retrieve job because the BLOB couldn't be deserialized: coldfusion.util.CaseInsensitiveMap; local class incompatible: stream classdesc serialVersionUID = -5469900980072247315, local class serialVersionUID = -4876788950396709071

[yuliangr39032863]

i have a gut feeling that issue is CF2016 serialized the scheduler data using JRE 8's Serializer class   whereas CF2018 is trying to deserialize that same data using JDK 11.  

[Charlie Arehart]

I realize this is nearly 3 years old, but I can clarify at least that:

a) this problem ALSO happens with an attempt to use in CF2021 a sched task cluster DB that was created and populated with CF2018 (so it's not a CF2016/2018-only thing, the versions originally indicated here)

b) this is also NOT a problem of JVM version differences (as yuliangr wondered in their last comment). In my observation of things discussed above, both cf2018 and 2021 were running java 11.0.14

c) FWIW, you don't really need to modify any code to see the error message that yuliangr found. It's tracked also in the cf exception.log file.

All that said, I have not yet been able to determine why it is that trying to use a given clustered sched task db across different CF versions doesn't work. Like others here, I presumed it would.

[w49369461]

Can confirm this is an issue with 2021 upgrade from 2018. CUrrently going thorugh this pain.

Have an active case open with Adobe we have been working through for nearly 2 months to try and resolve this but still not luck.

Had the same issue at Christmas when the Log4J issue came up. The patch for 2018 killed the clustered scheduled tasks in the same fashion. Adobe were able to provide a hotfix that resolved that issue but their attempt to use a similar hotfix to resolve this issue with 2021 failed.

Hopefully they will get back with a resolution and I will share anything I can

[Charlie Arehart]

I can share some more info. After digging in considerably, I think I can say confidently that the problem is not so much that the CF versions differ but that the underlying quartz libraries in the two CF editions differ. And that's likely been the case for each CF release, since the quartz-based scheduled task mechanism was added to CF over a decade ago.

More specifically, I tried various db compare tools (that was fun--not) and found that the difference is in a serialized chunk of data stored in the qrtz_job_details table (for each task) as the job_data column. This seems to be related to what's referred to in the error as "local class incompatible: stream classdesc serialVersionUID").

As I'd mentioned, I find folks in other apps (that use Quartz) complaining of the same error, and I found one saying they found it to be about the different quartz versions between their two implementations--and its impact on that same serialVersionUID. That's when I realized this would seem reasonable to be the root cause problem for us.

One of them even talked about how a way they worked around the problem was to export all the data except that column, and then they re-serialized it. It's not clear whether and how we could use what they offer--not only because they offer Java code that is specific to the app and related components they were using, but also because we don't see the java code that underlies CF to do the serialization in the first place. 

(And recall that I'd mentioned above how I'd already confirmed that my cf2018 and 2021 were the same Java version, so that's not where the problem was. And I'd created the task and indeed the db--and told cf2018 to create the clustered task tables only this week, so the jvm is not the issue.) 

So, bottom line, I don't yet see a solution. But at least there's light at the end of the tunnel. Maybe these ideas may help spark thoughts by others. (It's also late for me and it's been a long day. Perhaps with a new day I will see something I'm not thinking of now.)

Finally, to your previous comment, w, I must say I find it hard to believe that the Dec 2021 CF update for Log4j could have had any hand in causing any error related to scheduled tasks. I do realize you believe it, and may feel even that Adobe confirmed it, but since that update only addressed a very small number of jars--and those specifically related to log4j, I just really don't see any connection.

Now, could you have had an error in the installation of the update, which had some OTHER unexpected impact? Absolutely. I see that all the time (and have blogged about it).  (To this discussion about tasks, I'll just say that since updates are cumulative they lay down all the changes from prior updates, so that if there's ANY failure during the update, it can break even things that were not "changed" in "the latest update".) 

Anyway, if you or Adobe learns more, please do share.  And I will try to do the same.

[Charlie Arehart]

Ok, after a night's sleep, I think I see a path forward. We (in this situation with CF) should be able to just use sql from the newer cf version (like in a cfquery, or whatever you prefer) to pull out the key data for each task, then use cfschedule (or the admin api) to ADD each as a new task (in the newer cf version), with a new name so as to not override the current (working) one.

Then we can compare them (or some test ones), and even set up code to then delete the old ones (once we're happy with the resulting migration) and it can then also rename the new ones to take their place.

I'll give it a shot and report how it goes. Or I look forward to anyone else's ideas. 

[Charlie Arehart]

So I have some more news, updating what I'd written this morning. Ya want the good news or the bad news?

Let me start with the bad news, to round out my comment from last night. Then as for the good news, there's are three other possible solutions (one with a gotcha) to "export" the clustered tasks from one CF version to another.

And as a TLDR, I am going to propose that what seems the best solution (to get clustered tasks from one CF instance to another) will be to simply use the good ol' CF archive or CAR feature. More to follow. 

1) First, after much digging, it's not as simple as I'd hoped (to pull the data out of the db). While there is SOME of the data about each task in the table cols (esp in the qrtz_job_details and qrtz_triggers tables), the data in the job_data col (in qrtz_job_details) is what is in a binary format which must be what is generated distinctly for each CF (or underlying Quartz) version.

As such, since Adobe doesn't document in anyway what is there and how it was created (or perhaps serialized), and I don't find it in any Quartz docs or resources, it seems this will be a dead-end for recreating tasks from the tables (because we only get a portion of what we need).

2) Second here's the "good" news of alternatives, and I wish I'd thought of them initially. Well, they will suffice for most folks--at least if you have still (or can install) the version of CF where you created the tasks that are clustered. At least what I'm about to offer are three ways one can "export" the tasks and then "import them" to the newer CF version. 

2a) The first may appeal as a way that would not require admin access (with a gotcha). As you may know, the cfschedule tag (or script equivalent) has an action="list" which returns a query with all the tasks for the current instance--whether clustered or not. I had implemented that to "check my work" while I was trying to extract the data from the db (running it on the cf version that had created the tasks).

Then I realized (after discovering point 1 above) that I could in fact just try to use that query resultset from the cfschedule action="list" to see everything I needed to recreate the tasks. (Again recall the original goal of how to migrate the clustered tasks from the earlier CF version into the new version.)

But there's a gotcha: some may know that since cf10 there's the notion of a task being defined either in "application" or "server" mode. The intent is that app1 can't see tasks for app2. Well, while there is a mode attribute to get tasks of each type with the action="list" attribute, beware that a cfschedule running in app a can't see the tasks created for app b. So that's not a great solution--for those whose admin shows that they do have app-mode tasks. (The admin shows all tasks.) But a probem is that perhaps you don't have access to the CF Admin--or won't be told what tasks it does or doesn't list.

So what if you do find in the CF Admin (or are told) that you have app mode tasks? Well, then you need a different solution. Both do require admin access, but for many that's not a problem.

2b) Of course, you can use the CF Admin "coldfusion archive" (CAR) feature, on the "packaging and deployment" page of the admin, to see and select some or all tasks to be exported, to a car file. You could then take that car to another server and import it into the other admin (yep, even a different CF version). 

This is another "why didn't I think of this sooner?" option. But in my defense I was thrown off because I'd already found that the cf2021 cfsetup tool (which can export settings) was not seeing the details of clustered tasks. And I knew that the neo-cron.xml file did not track clustered tasks. So I fell into a trap of thinking "the info is only in the clustered db". When I hit the serialized/binary job_data col that I couldn't "break open", I was forced to think "outside that box".

2c) Finally, I mentioned that there were 3 alternative solutions. Again, once I got outside the box of the db, I realized that a variation on my point 2a above (using cfschedule to get and recreate the tasks) was that one could just use the CF Adminapi. It's got a scheduler.cfc with methods to do the same. But really, why bother with coding such processes by hand when the CAR mechanism would be the simplest.

Indeed, once the client I was helping understood these options, they said they would take it from there. So I never created the code for my point 2a above.

But I leave the ideas (and the discussion) in case it may help others facing the challenge in the future.

[w49369461]

Should have been clearer. In patching for Log4J we were already a hotfix or two behind, as its cumulative, it installed these too causing the issue.

As the HF are cumulative, there were other things updated to the libs that caused the same issue we see now when we patched for Log4J.  Unfortuantly their method of resolving the issue then didnt work for upgrading from 2018 to 2021

Those servers I manually patched with just the Log4J libs did not have the issue.

[Dave Watts]

This gets a "yikes" from me. I'd recommend that you rebuild your schedule list in the OS scheduler, or follow @Charlie Arehart 's advice and export from the old format to the new one.

Dave Watts, Eidolon LLC

[Charlie Arehart]

w, yes for sure, the other reason things can seem not to work is when the update installs fine, but it (or some one you're skipping) implements a change that leads to trouble. In the case of the Dec 2021 updates (3 for CF2021, 13 for CF2018), there was no change but the log4j mitigations.

Conversely, there were bugs in the Sep 2021 updates (2 and 12, respectively) which REMAIN to today, and there are fixes from Adobe for many of them, but which won't be made available to everyone until the next update (which I hope is any week now, as it's been 7 months of trouble since those Sep updates introduced those bugs).

So sure, you could have a problem caused by going to the Dec update having skipped the Sept one. And if instead you did the manual mitigation to address log4j (INSTEAD of the Dec update), that would avoid you implementing the changes/bugs in the Sep update. But beware: the manual updates Adobe offered were SPECIFICALLY indicated to be used for a post-Dec updated CF instance. They were NOT intended to be used INSTEAD of the Dec updates. I get it that it seems like it "worked" for you. The better news is that there are no known instances of the log4jshell issue affecting CF, so that was all a lot of hubbub for nothing--and you seem to have lucked out.

But all this is quite separate from the real focus of this thread, which is the issue of using the clustered task db from an earlier CF version in a later one. I have more to share on that, and will do that in a separate reply.

[Charlie Arehart]

I'll add one more (perhaps last) thought on the matter of exporting clustered sched tasks from one version to another. I'd suggested that the car mechanism may be the most effective.  Two more points about those. 

I forgot to mention that the Adobe CF docker images are designed so that if you put the car file/s in the /data folder of the image, it will be auto-imported at container start. 

Second, the client I was helping had been interested in the idea of having a json file with the task definition (to reuse in various ways). Well, note that the car file is just a zip with an xml file holding the tasks (or whatever admin info is exported). So it would be fairly trivial to have something convert that xml to json. 

(Recall also that I'd shared how the the cf2021 cfsetup tool did not export the clustered sched tasks. I've not had a chance to see if the Commandbox cfconfig tool does.)

I may be tempted to dig into things still more, but in case not, again I leave this for others who may find it. 

[BKBK]

i did a tiny bit of CFIDE hacking.

edited \cfusion\wwwroot\CFIDE\administrator\scheduler\scheduleTasks.cfm

line 158 in the try catch block to push the exception message to efrErrorMessage

By @yuliangr39032863

Bad idea. Strongly discouraged.

You should not tamper with ColdFusion's engine.

[BKBK]

the clustered scheduled tasks was initialized with CF2016

we try to go to CF2018's CFIDE Scheduled Tasks and attach this new node to the existing datasource for the cluster that has existing running schema.

is it possible CF2018's clustered scheduler is not compatible with CF2016's clustered scheduler?

By @yuliangr39032863

Yes, it is possible that scheduled tasks created in ColdFusion 2016 may be incompatible with ColdFusion 2018. Likewise, scheduled tasks created in ColdFusion 2018 may be incompatible with ColdFusion 2021.

The reason for the possible incompatibility is: transferring scheduled tasks between major ColdFusion versions is undocumented functionality. As such, ColdFusion cannot guarantee backward-compatibility. One reason for this is the evolution of ColdFusion. 

For example, ColdFusion version X+1 may introduce a fix or improvement that affects scheduled tasks. Such a change may be absent from ColdFusion version X. So, even if you succeed in transferring scheduled tasks from CF_X to CF_X+1, there will be a risk of malfunction or error. That is perhaps what @w49369461 found.

The solution is, of course, documented functionality. The ColdFusion Team should include a new feature enabling scheduled tasks to be transferred from version X to version X+1. 

[Charlie Arehart]

Bkbk, just to clarify, when you refer to "transferring" tasks between versions, note that this thread is NOT about the old miatake (that many do indeed still make) of copying the neo-cron.xml file among cf installations.

Instead, this was about someone who simply had the same db defined as a dsn for both cf versions, and that dsn had been used with the cf admin feature to store clustered scheduled tasks.  They're finding THOSE tasks do not appear in the newer cf version.

Perhaps you've read the other replies where I offer info on how to deal with that. As for Adobe offering solutions, I point to a couple (car, cfsetup, etc.) Each will suit some folks bettet than others.

But I'd argue that the ultimate solution would be if the newer cf version could "just handle" somehow being able to use the same db holding the previously defined clustered sched tasks, from at least the immediately preceding version. I suspect there are reasons we may never get that solution. 

[BKBK]

Charlie, I read all the replies and took them into account. I was talking about the subject of this thread. That is, about the transfer of database-stored, clustered scheduled tasks.

Such clustering is tightly-coupled to the scheduler. In fact, arguably more so than ColdFusion's use of the neo-cron.xml file. The underlying scheduler engine may change between major ColdFusion versions. Hence, if you carried over undocumented clustering functionality from one major ColdFusion version to another you would be taking risks. There might be breaking-changes or errors. Even if the carry-over worked today, it still might break tomorrow after ColdFusion implements a hot-fix.

Unless, of course, the ColdFusion Team implements a new feature to transparently enable the transfer of clustered scheduled tasks.

[Charlie Arehart]

I don't see anyone proposing to "carry over undocumented clustering functionality",  per "the subject of this thread". 

But this is another one of those where we seem at odds, which serves no one to prolong. You do you. 

[BKBK]

I don't see anyone proposing to "carry over undocumented clustering functionality",  per "the subject of this thread". 

By @Charlie Arehart

Subject of this thread: "2016 to 2018 migration with clustered scheduled tasks".

That is what I mean by carrying over undocumented clustering functionality from one major ColdFusion version to another.