Schedule Task not working after 2021 Coldfusion Upgrade

[VinoD22934182q2do]

Hi,

I have Coldfusion installed on linuxOS  (as a code) on AWS instance. Trying to upgrade Coldfusion from 2018 to 2021 with the same working setting/code.

After upgrade, Schedule tasks is not working. When tried to manually run a task with output file, it doesnt run at all, and no logs are captured on scheduled tasks.

Scheduler package has been reinstalled, still not working.

In http logs, could see entries - which means it is not permission issue.

And the URL used in the task is HTTP - so not key issues.

Any idea on this issue ?

[Charlie Arehart]

While it's understandable that you feel the problem is with the scheduled task mechanism, you should at least try the url in a cfhttp running on the same cf instance. Then dump the resulting cfhttp variable. What does it show? And how long does it take?

There can be various reasons why a call out of cf could vary between a cf2018 and a cf2021 instance, even from the same server. There could be jvm differences, beyond the differences in cf itself. 

In fact, what jvm do you see cf running on in each cf version? It's on the cf admin settings summary page, for example. And what cf2021 update are you on (same page)?

(Yet another sanity check would be to setup a new task going to a url that you know works from the browser on the server.) 

[VinoD22934182q2do]

Hi Charles, I am delighted to see your response. Thank you!

I havent tried the first step, however below is the jvm versions and the update version used.

cf2021

Server Product ColdFusion 2021
Version 2021,0,03,329779
Update Level /opt/coldfusion/cfusion/lib/updates/chf20210003.jar Java Version 11.0.11

cf2018

Server Product ColdFusion
Version 2018,0,13,329786
Update Level /opt/coldfusion/cfusion/lib/updates/hf201800-0022340.jar Java Version 11.0.11

And i access the same url in the cf server, i am able to access it. It is just that the task doesnt run at all, if the url is not working i would see the same errors in the logs, isnt it. In my case, the url is working when access from the server, and i dont see any logs for error.

I would check on your first step and update the status.

[Charlie Arehart]

Thanks for the update, Vino. And first, please call me Charlie. 🙂 Only my sister and a nephew still call me Charles. Long story.

So besides my first suggestion of testing a cfhttp call to the task url (very important), note also my last one of testing a task calling something else, like simply google.com. Let us know how those go. 

[BKBK]

As there is nothing in the logs, it implies the tasks are not running. It just might be that the tasks are not yet fully registered on CF2021.

To verify, open the CF2021 Administrator and go to the page Server Settings > Scheduled Tasks. Are the tasks registered there? If so, do they have the settings you expect? Verify that each setting is valid.

Do some simple tests. For example, what happens when you click on the button to run a task? Also, the edit button enables you to edit a task where necessary.

[VinoD22934182q2do]

The scheduled task are definetly present. I see them in the console. I had also created some test tasks, and tried it running, it never runs.

[BKBK]

Weird that ColdFusion fails to show any errors.

You could try:

1. Upgrading CF2021's Java to 11.0.15;
2. Going to the ColdFusion Administrator Package Manager > Packages page, clicking on Available Packages, and ensuring that all packages are installed;
3. Doing the following test
   (Place testScheduler.cfm and testTask.cfm in the web root (wwwroot). Then launch the URL of testScheduler.cfm in the browser.)
   

   <!--- testScheduler.cfm --->
   <cfschedule
    action="update"
    task="testTask"
    operation="HTTPRequest"
    startDate="#now()#"
    startTime="7:00 AM"
    url="http://localhost:8500/testTask.cfm"
    interval="60" />
   ​

   
   

   <!--- testTask.cfm --->
   <cfscript>
   testString="The time is now: " & dateFormat(now(),"full") & " " & timeFormat(now(), "HH:nn:ss");
   writedump(var=testString, format="html", output="#server.coldfusion.rootdir#/logs/scheduledTaskTestResult.html");	
   </cfscript>​

   As a result the file scheduledTaskTestResult.html should be created in the /cfusion/logs directory. Verify.

[VinoD22934182q2do]

Hi

Could you please share me the link where i can download the jvm 11.0.15, i could see only 11.0.14 from the adobe downloads. Infact, i upgraded the jvm to 11.0.14 and tried with the steps in created scheduler and tasks. When accessed the url, no logs files got generated. I only see the below logs updated during the same time when accessed the testTask.cfm

Update.log
"Error","ajp-nio-127.0.0.1-8020-exec-3","05/10/22","18:56:10","CFADMIN","Not able to connect to Update Site: Connection Failure: Status code unavailable"
Application.log
"Information","ajp-nio-127.0.0.1-8020-exec-3","05/10/22","18:56:10","CFADMIN","Connection Failure: Status code unavailable"
httpd.log
"Information","ajp-nio-127.0.0.1-8020-exec-6","05/10/22","18:56:10","","Starting HTTP request {URL='https://www.adobe.com/go/coldfusion-updates', method='get'}"

In the meanwhile, please share me the jvm download for version 11.0,15, i would try with that version as well....

And one more thing which i see when comparing with working cf version (2018) and non working cf version (2021) in java and jvm section. Below is the class path updated in 2021 version, could you please review and update if anything missing here

{application.home}/lib/updates,{application.home}/lib/,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/cfform/jars,{application.home}/bin/cf-osgicli.jar

[VinoD22934182q2do]

I see this log as well in exception.log (removed project specific details)

"Error","ajp-nio-127.0.0.1-8020-exec-4","05/10/22","18:53:07","","File not found: /wwwroot/testScheduler.cfm The specific sequence of files included or processed is: /opt/coldfusion/cfusion/wwwroot/wwwroot/testScheduler.cfm'' "

coldfusion.runtime.TemplateNotFoundException: File not found: /wwwroot/testScheduler.cfm

        at coldfusion.filter.PathFilter.invoke(PathFilter.java:165)

        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)

        at coldfusion.filter.LicenseFilter.invoke(LicenseFilter.java:30)

[BKBK]

I see this log as well in exception.log (removed project specific details)

"Error","ajp-nio-127.0.0.1-8020-exec-4","05/10/22","18:53:07","","File not found: /wwwroot/testScheduler.cfm The specific sequence of files included or processed is: /opt/coldfusion/cfusion/wwwroot/wwwroot/testScheduler.cfm'' "

coldfusion.runtime.TemplateNotFoundException: File not found: /wwwroot/testScheduler.cfm

        at coldfusion.filter.PathFilter.invoke(PathFilter.java:165)

        at coldfusion.filter.IpFilter.invoke(IpFilter.java:45)

        at coldfusion.filter.LicenseFilter.invoke(LicenseFilter.java:30)

By @VinoD22934182q2do

The directory /wwwroot/ occurs twice. I expected it to occur just once. 

[BKBK]

Could you please share me the link where i can download the jvm 11.0.15, i could see only 11.0.14 from the adobe downloads.

By @VinoD22934182q2do

 Adobe was probably working on the downloads page. Here it is: https://helpx.adobe.com/coldfusion/kb/coldfusion-downloads.html 

[BKBK]

And one more thing which i see when comparing with working cf version (2018) and non working cf version (2021) in java and jvm section. Below is the class path updated in 2021 version, could you please review and update if anything missing here

{application.home}/lib/updates,{application.home}/lib/,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/cfform/jars,{application.home}/bin/cf-osgicli.jar

By @VinoD22934182q2do

I can confirm that that is the value of the flag -Dcoldfusion.classPath on my CF2021 installation

[BKBK]

@VinoD22934182q2do , any update on this? Are your scheduled tasks now running?

[Bill Tudor]

Hi Charlie,

thought I'd use this post to resurrect an issue with <cfschedule> as we are experiencing some very weird behaviour using CF2021 to upgrade some CF2016 code. We are finding that the cf Admin is not getting updated by the <cfschedule code nor is a manual entry of a schedule via the Admin 'working' in the sense that the Admin tells us it is completes successfully but the log reveals this:

"Error","DefaultQuartzScheduler_Worker-2","08/17/23","09:45:30","","Connection Failure: Status code unavailable "

Basically we seem to have issues with whatever Quartz is!

This is on a Windows22 machine,iis,CF installed via the GUI_64, using the bundled CF jre (i.e. nothing fancy)

Your help really appreciated as this is blocking our upgrade path.

Cheers, Bill Tudor (we met at CFCamp this year after your presentation)

[BKBK]

@Bill Tudor ,

While you wait for Charlie's answer there is something you could do. Perform the cfschedule test I suggested on May 09, 2022. Then share the result.

[Bill Tudor]

Will do, all suggestions here are welcome and will be used to track down various issues we were having, but in the interest of sharing we found that we had a class conflict at the Quartz level due to incorrect AppDynamics javaagent being loaded. Once we got the right version of that for CF2021/JVM 11 the scheduled tasks kicked off OK. Seems AppDynamics also uses the Quartz based scheduler and somehow this must have been breaking CF. As you can imagine this was quite hard to track down as diagnosing the server logs is tricky when all you see is a Quartz Error with something like "Status Code is unavailable". I guess there may well be a way to get more verbose output, but at the time that's pretty much all we had to go on.

However we have noticed other issues with sheduled tasks as yes we are also porting them across from 'old' to 'new' servers so will be looking into password issue too as mentioned by @boopower (thanks, could well be something we need to look into). And thanks Charlie for response as always.

[Charlie Arehart]

Bill, see what I proposed as a diagnostic in my first reply here . Consider also other tips I shared in my preso on scheduled tasks last month, available at carehart.org/presentations.

Let us know what you find. 

[Bill Tudor]

Charlie, yes we tried running the scheduled task url outside of scheduler (that was fine), it was only breaking inside the scheduler. The behaviour didn't make much sense until we finally tracked down the AppDynamics problem, as I posted above, which was a hrd one to find! Regards, Bill.

[Charlie Arehart]

Thanks for the update, Bill. Yes, that was a tough nut for you to crack--and one of the rare times that a request would fail as a task but NOT as a cfhttp call from the same instance. Good on you for sorting it out.

(BTW, there is indeed additional logging one can enable in quartz that may have helped. I've been meaning to do a post on that.) 

[Bill Tudor]

yes a tough one to crack; and to be fair, not necessarily an Adobe/CFML issue as such. A colleague mentioned that you can even update quartz 'inside' the JVM independently, but we don't want to hve to go there just yet! Extra logging from Quartz could be useful. We do have mission-critical scheduled tasks, which appear to be running ok now, but not fully tested yet in Production with our CF2021 upgrade.

[Charlie Arehart]

Something to beware about updating the quartz version within cf is that of course you could then break some aspect of cf scheduled tasks.

So do at least try always to remember to revert that before reporting a seeming "bug"--and also to mention such a change when asking for help in the community. 🙂 

[boopower]

The suggestion that the user gave for finding out what processes are running is brilliant.

<cfset arySchedTasks = createobject("java","coldfusion.server.ServiceFactory").getCronService().listall() />

Using this, I found that when I moved the scheduled tasks from one server to another by copying the neo-cron.xml from one server to another many of the processes stopped processing. When I reviewed the processes using the scheduled task list, I noticed that the processes that WERE working didn't have encrypted passwords in the password field while the ones that DID NOT work had encrypted passwords that prevented the processes from running due to the fact that when the processes ran, it couldn't decrypt the passwords that were in the neo-cron.xml file (as they were from a different server and therefore the decryption didn't work). To solve this issue, I opened up the neo-cron.xml file and manually took out all of the passwords in the password section of each profile and then saved it and replaced the file with the passwords in it and then restarted the CF service. Once restarted, the processes started running again by themselves.

I wanted to share this in case anyone else was having the same issue. If your tasks are kicking off after moving servers, check to make sure there aren't any passwords store in the neo-cron.xml file.

It took my ages to figure this out, so hopefully this helps someone else.

[Charlie Arehart]

Boo, do beware of that approach of just copying neo-*.xml files from one cf instance to another, not only for that reason but ESPECIALLY if the cf versions differ at all (version or update) as the xml could differ between them, causing subtle bugs.

One may wonder how then to transfer such admin settings between instances (other than manually retyping them in the cf admin). There are a few ways--and note that they ALL taken care of the decryption/encryption of passwords between the instances:

• The old gui-based CAR (ColdFusion ARchive) export/import feature, found in the cf admin (which was Enterprise-only prior to cf11 but was opened for all for nearly 10 years now--but many still don't know)
• The new CLI-based cfsetup tool added to cf2021, which allows export/import via JSON files. You can find more about by googling that word, including a substantial Adobe doc page as well as a webinar on it that I did. While the docs say not to use it with other than cf2021 or above, I show using it with as early as cf10 in the talk. It also backs up what it changes, which is nice.
• There is also the Commandbox cfconfig tool, which predates cfsetup, as another means to export/import settings between instances vian json. It has more features and flexibility, and I discuss it in my talk linked to below. 
• Of course, the first launch of the cf admin will offer to migrate in settings of a an earlier cf version found on the same machine. And that can be triggered to run again later, as I discuss in a blog post of also several years ago. 
• Finally, if one feels they have no choice but to do a "manual" migration, typing settings from one instance into another, there's at least another way to "check your work". I have a talk on migrating (or comparing) cf admin settings , and in particular I discuss an approach to automate comparison of admin setting in the two instances. I show using the output of the cf admin "settings summary" screen with a good compare tool (like BeyondCompare), which some may find helpful. The talk also discusses the other tools and processes above

I know this is all going off-track from Bill's issue here, but since you offered this approach of copying xml files (and he commented today he'd be checking if that may have happened), I thought I'd offer this for you both or other/future readers. 

[Bill Tudor]

Sometimes you may only have manual migration presented  as an option - in our case it's due to reliance on terraform code to build CF AWS instances where one step involves copying neo-*.xml. Without rewriting iac code it's hard to get around this. And that code is not necessarily accessible to all. Having said that, I have noted your talks around this subject and will be reading up: particulalry with regard to  "the xml could differ between them, causing subtle bugs." Thanks for raising awareness on that.

[Charlie Arehart]

I appreciate that challenge. But when it comes to such automated ops/config, sometimes old approaches should be reconsidered when new ones come along.

And I didn't clarify in my reply that the cfsetup tool enables export/import of admin settings via json. I've added that.

I also added mention of Commandbox cfconfig, as another such option. I also clarified that all the options take care of decrypting/encrypting passwords in admin settings.