Tasks & SSL

[dtb26]

Hello,

I have a long standing issue with SSL via tasks that has plagued me throughout CF10, 11, and now 2018.

When we initially stand up the server, we generate the certs and add them to the keystore; everything works great.  Fast foward a year, the cert expires and you update it; all tasks break with "Connection Failure".

I have tried:

- adding the new cert and restarting (continues to fail)

- deleting the inappropriate certs and then adding the new cert (continues to fail)

- searching the box for any cacerts store and adding the right cert to those (continues to fail)

- giving up on the newly gernerated crt file, I remotely connect and generate the cert from the remote site and then add it to the castores (its seen as a new cert, but connections continue to fail)

The fact that I can set this up during the initial install tells me my process is appropriate.  the fact that CF can never again see the cert after an update makes me think theres a cache or somthing somewhere that needs to be purged.

has anyone else ever run into this; possibly have some guidance to point me in the right direction?

Thank you

Dave

[Charlie Arehart]

CF updates don't touch the keystore within the jvm that CF uses, so when you say things break after an update, can you be more specific: are you really saying that ONLY CF is updated? not also the JVM that CF uses? 

If you look at the cf admin "java and jvm" page, what is the current "java home" (or the value of the same name in the jvm.config)? If you have updated CF to use a new jvm, then if you do NEED to import certs, you would need to do that into the cacerts under the lib/security folder of that JRE that CF is pointing to.

That certainlty had tripped people up: they update their CF JVM, and find that something no longer works, and they read instructions showing how to use the keytool--and they are updating the wrong one. I know you said you looked for "any". Let's have you look at the one in the JVM that CF is pointing to. Is the cacerts file in that lib/security a different date and time than all other files in that folder? If not, then that cacerts never had any certs imported into it.

Finally, perhaps you don't NEED to import certs. Perhaps instead all you REALLY need to do IS to update the JVM that CF uses. I have a blog post on this, with more detail:

https://coldfusion.adobe.com/2019/06/error-calling-cf-via-https-solved-updating-jvm/

Let us know if any of this helps get you going, or not.

[dtb26]

Hi,

The only update I was referring to is the cert itself: 

"Fast foward a year, the cert expires and you update it";  "it" being "the cert".

> Let's have you look at the one in the JVM that CF is pointing to. 

I attempt to update the store that I initially configured at install time; ([cfroot]/jre/lib/security/cacerts) coldfusion just doesnt seem to see the new cert.  Out of desperation, I have executed "find / -name cacerts" and then add the cert to all of them (coldfusion comes with two, one under the jetty file structure).

> then that cacerts never had any certs imported into it.

I'm able to verify that the certs have been imported by using the -list switch; I've no doubt that they are being imported.

But again, this all works fine with a new install, its just updating the expired cert that seems to be a problem

thank you for the response.

Dave

[collaboranaut]

You mention cacerts, did you check keystore.jks ?

[dtb26]

I did not; I'm completely unfamiliar with that... I'll research.

thanks!

[Charlie Arehart]

I appreciate you are being clear and precise. I will repeat a question you did not specifically answer:

• If you look at the cf admin "java and jvm" page, what is the current "java home" (or the value of the same name in the jvm.config)? If you have updated CF to use a new jvm, then if you do NEED to import certs, you would need to do that into the cacerts under the lib/security folder of that JRE that CF is pointing to.

I realize you did the "find" and that should cover "all possible cacerts". Please let's just confirm things: what jvm is CF using? and does the cacerts in that folder's lib/security show a date different than other files in that same folder? That's really what matters.

We can talk about what to do next based on your answers (which can vary depending on the CF java location or the cacerts date, and more).  Again, the CF updates themselves do not touch the cacerts. I realize you feel that's "all that has changed", but unless I'm mistaken there seems to be some other explanation for what you are experiencing. And I help people solve these sort of problems daily, so we should be able to solve this one for you.

[dtb26]

Hello,

Thank you for the follow up...

CF JVM is:  [cfroot]/jre/

Cerificates installed to:  [cfroot]/jre/lib/security/cacerts

and just to cover all bases, also added the cert to: [cfroot]/cfusion/jetty/jre/lib/security/cacerts

In all cases I could use the following to verify it installed:

keytool -list -v -keystore [path]  -alias [target alias]

I verified the date on both:
cfrun bin 96752 Jun 16 10:39 cacerts
and
cfrun bin 90395 Jun 16 10:07 cacerts

> I realize you feel that's "all that has changed"
honestly, this has been a recurring issue since Coldfusion 10.  Theres a lot to be said for your theory of the java implementation needing to be updated; especially if we had updated OpenSSL or if we had altered an Apache config to limit ciphers and/or protocols.  However, in this case, it really is as simple as: everything is running normally monday night, tuesday morning we update the cert, all tasks break at the point of update.

Thanks again

Dave

[Dave Watts]

This may be a dumb question, but are you updating the entire certificate chain? I've run into several cases where the certificate is coming with a different chain than it did before. Maybe there'll be new intermediate certificates, sometimes even the name of the root certificate is different. And these are less likely to be in your cacerts file as they're typically CAs that have been recently created.

Dave Watts, Eidolon LLC

[dtb26]

Thats a great question.

In this particular case, our site cert hadn't actually expired yet; the USERTrust intermeadiary had, so I did add alias' for both the site cert and the full bundle.  We have a couple people that connect to the machine programatically and their code was being very strict about validating the full chain.  It might also be worth noting that I added the diffie-hellman parameters fix as well.
Other than the coldfusion scheduler, everything appears quite happy with the updated cert (in point of fact, my current workaround to the scheduler issue is to script curl calls and cron them).

thanks for the response

Dave

[Charlie Arehart]

Dave (dtb26), sorry that I kept somehow misunderstanding in thinking that you were saying the problem was when you updated *CF*. I see now (especially from your last reply to me) that you are not updating CF at all, just the cert. (And I see you clarified that in your first reply to me. I just was somehow blind to it.)

So I get it now, but let me restate things for myself and perhaps to benefit others following alone: your issue is that a cert expires (on a server that a CF scheduled tasks tries to talk to), and so the scheduled task fails. But then you update the cert (first on the server running the page you are calling in the task), and yet the scheduled task still fails. And then you update the cert WITHIN CF, and again the task STILL fails.

And that's why you wondered if somehow CF is somehow "caching" something.  But you clarified in the first note that you even restart CF and that does not fix it. So you find that once the cert is updated, NOTHING allows CF to talk to that server via a call as a CF scheduled task, right?

And I'll say I have never heard of that before. And I suspect you have googled and googled for info on it and found none, which is why you're here.

OK, so let's talk about some things to consider, first two forms of debugging you could add, and then two alternative approaches you might consider.

First, can you set up a CFHTTP call to the same url (as the task) and run that from the CF instance, and cfdump the cfhttp scope. Is there any more useful info there, vs what you may have gotten from the failed sched tasks (or logs)? What IS the error you see, in the result from the cfhttp? To be clear, all the scheduled task does is the equivalent of what cfhttp does, so both should fail for the same reason.

Second, if that answer doesn't somehow give us a clue, the next thing to consider is logging debug-level info about the handshake that CF (the jvm) makes with your destination server, whenever either a scheduled task or cfhttp (or similar tag) makes a call to an https url. All you need to do is add a new jvm arg to the java.args line in the CF jvm.config:

-Djavax.net.debug=ssl,handshake,verbose

On restarting CF, the next time you make any such ssl call out of CF, there will be many log lines in the coldfusion-out.log file, which you may be able to understand and perhaps find the answer. If not, you can share it here (but please do that as an attachment, rather than dumping it in the body of a message.

If somehow you continue to stuggle, I'll point out a couple of other options to consider.

One is that besides your using curl for now, yet another option to consider is the commercial cfx_http5 alternative to cfhttp. No, it's not free but it's not expensive, and some have found that it has solved problems where a cfhttp failed. And no, it won't be used somehow by the CF scheduled task mechanism, but you could change your task to call a CF page that uses the cfx to then call the url to be run. I realize it's awkward, but I'm just saying it's an option.

The other is that again you may want to consider updating the JVM. Even the one in CF2018 out of the box is 2 years old. See the blog post I shared about how to go about updating that. It can be  EASILY reverted back, if you wanted to do taht for any reason. And I'd be curious if it worked first WITHOUT any need for you to import certs into the cacerts of the new jvm. If it did, then that's one less hassle for you to worry about.

Again, I realize that even if it DOES work, you will be left wondering if a later cert update (on the destination server) would break things again.  It really should not, but at least updating the JVM will be good for other reasons (security, bug fixes), until that happens (which would happen also for you with the old jvm, if we don't find another solution.)

So those are some ideas. Hope one of them helps.

[dtb26]

Hi Charlie,

Thanks for the verbose response; there is a lot for me to work with there.

Your statement of the issue is nearly spot on.  One point of clarification:  the task actually completes successfully from coldfusions perspective.  The response and the logs all indicate a successful "task".  The target of the task is never executed though.  If I redirect the output to file, the result of the task is a single phrase:  "Connection Failure"

This is where Google fails me as I get buried in a sea of people posting about how the error means the handshake cant be completed and you have to add the certs to the keystore.

> And I suspect you have googled and googled 
literally for years (off and on).  I first ran into this under CF10, then CF11, and now CF2018... decided I'd like to just figure this out once and for all 🙂

I have some work to do.  I'll post back with results.

thank you again

Dave

[rahmed09]

hello,

there's some good info on this page about this issue, but no resolution. Were you able to fix it please? if yes, will you please sare the resolution 🙂 . Thank you!

[dtb26]

Hi,

I feel like Adobe may have quietly adressed this in one of the 2018 patches as it automagically ceased to be an issue.  Our server in question runs apache vhosts so the cert is updated more than once a year, and so far, CF Scheduled Tasks is seamlessly picking up the new certs.

Prior to this, I never found a fix.  The workaround was to employ Curl or wget via cron (those never had a problem with the cert updates).

Thanks

Dave

[rahmed09]

well, we've started seeing the same issue on CF2016 where the Scheduled tasks works without any issue when pointed to older a bit older JDK "jdk1.8.0_321", but with anything above this, I get the following error in the logs..

"Information","DefaultQuartzScheduler_Worker-1","10/24/22","11:17:56","","Task DEFAULT.NGA Notifications triggered."
"Error","DefaultQuartzScheduler_Worker-1","10/24/22","11:17:56","","Connection Failure: Status code unavailable "

I've tested with these JDKs (jdk1.8.0_331, jdk1.8.0_341, jdk1.8.0_351). i'm totally stumped at this point...

[rahmed09]

here's the full error from exception.log:

"Error","DefaultQuartzScheduler_Worker-2","10/24/22","11:29:16","","Connection Failure: Status code unavailable "
coldfusion.tagext.net.HttpTag$HttpConnectionFailureException: Connection Failure: Status code unavailable
	at coldfusion.tagext.net.HttpTag.connHelper(HttpTag.java:1332)
	at coldfusion.tagext.net.HttpTag.runCall(HttpTag.java:1434)
	at coldfusion.scheduling.CronTask.execute(CronTask.java:124)
	at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
	at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)

[dtb26]

Hi,

I don't think thats quite the same issue; our problems arose when updating the SSL certificate.  In an apache vhost environment, if you want to add a new vhost, the certs need to be regenerated to include the additional "alternative".  Your issue does seem to be presenting the same way, but I'm not sure the root cause is the same.  IF it is the same issue, there was no solution until mid-2018 and we had to employ a workaround using curl and cron.  If youre running from a windows box, I'm not sure what the equivillent would be; something in powershell perhaps?  Sorry I cant be more helpful.

D

[rahmed09]

Thanks I appreciate your help!

the issue got resolved by adding the following line in JVM arguments using the CFIDE.

-Dhttps.protocols=TLSv1.2

Its might not be a secure solution. but for that I believe we'll have to import the cert which our application uses into the JDK itself...

[Charlie Arehart]

I'll offer here that rahmed had raised this in another thread, where I offered very different information to consider. If someone finds this here and it doesn't work (or seem "right" to add that arg), please see that other thread:

https://community.adobe.com/t5/coldfusion-discussions/coldfusion-2016-scheduled-tasks/m-p/13296015