CF 2023 Update 15/16 trouble with encryption keys

[moalspvic]

We are running CF 2023 and just updated from Update 14 to Update 16.  

We have two errors that are occuring, because of the reference to Bouncy Castle, one of them looks like the CFMAIL error  that others experieneced, however clearing the felix-cache and restarting the instances did not work for us.  In both cases, we are using encryption keys and the code base hasn't changed and worked previously on Update 14.

1) trouble with <cfftp>

<cfftp action="open"
            server="sftp.********.com"
            port="22"
            username="********"
            key="C:\********\privateKey.ppk"
            passphrase="********"
            secure="yes"
            connection="connFTP" />

This call is now returning the following error message.

struct
Cause	'argon2' is required, but it is not available.
Detail	Verify your connection attributes: username, password, server, fingerprint, port, key, connection, proxyServer, and secure (as applicable). Error: 'argon2' is required, but it is not available..
Message	An error occurred while establishing an sFTP connection.

2) We are using a utility to manage webPush requests through a web service and have to bind our keys to the object

<cfset PushService = createObject("java", "nl.martijndwars.webpush.PushService").init(variables.publicKey, variables.privateKey, "mailto:​x​xxx@xxxxxx.xxx")>

This call is now returning the following error message.

- Type: java.security.spec.InvalidKeySpecException > Undefined

- Type: java.security.spec.InvalidKeySpecException

- Diagnostics: key spec not recognized null

[moalspvic]

I've reached out to Adobe for assistance with this as well and response has been slow.  Their initial advice was to add -Dcoldfusion.sftp.enable-sha1-algorithms=true  to the jvm.config, stop the service, clear felix-cache, restart.  This did not help address either issue.  

I tried to debug on a DEV machine, however am getting the same results.  I installed Update 17 locally and received the same error.  I am not sure what was changed by the update to 15/16, but it has broken the two processes where we use keys to authenticate a connection (ftp and pushService).

We have not rolled back to Update 14 yet, as we'd like to go forward and figured leaving this in place would be the best way to troubleshoot and resolve the issue(s).  However, we will have to rollback (and hope that resolves the issue) soon as we need to have the connection to the pushService operational again soon.

Any thoughts?  Thanks in advance for your help.

[moalspvic]

..

[Charlie Arehart]

Without a simple test case we can attempt, we're in a tough spot to confirm your problem. But before concluding the bug is in something changed IN the update (any of the 3), have you confirmed first (in any of the failing instances) that:

• There are 0 fatalerrors in the log for the update
• The downloading of packages tracked near the bottom of that log was successful
• And on the startup of cf following the update, do you see it showing uninstalling the packages updated and then starting each package, without error

If errors happened any of those places (which could be unique to you for any of various reasons), then the problem is not in the update but in the failed implementation of the update. It can happen, silently (if one does not check these logs after each update).

I cover this in a bit more detail in my blog post about each update, including the one yesterday, here.

Hope that's helpful. 

[moalspvic]

Charlie, 

Thanks for the information.  On first pass, I was wondering if anyone else had seen a similar issue or had thoughts on how this may be similar to the CFMAIL issue that was introduced with Update 15 when mail needed to be signed by a key/cert.  I thought perhaps similar, yet different, because we are experiencing like problems when trying to use keys/certs for authentication/signing.

I followed the steps outlined in the post you referenced and it doesn't appear that there were any issues with the application of the update. 

• From the update log
  
  • Installation: Successful.
  • 1222 Successes
    0 Warnings
    0 NonFatalErrors
    0 FatalErrors
• All packages were downloaded successfully and copied into location for each of the 3 instances.
• There is a long list of packages being unistalled, then a list of packages starting...without error

Since you have not experienced either of these errors, what additional information can I provide that would assist with troubleshooting this?  

I appreciate your help.  Thank you.

[Charlie Arehart]

Thanks for confirming. So if that is indeed so, I'm afraid I have nothing more I can propose. You may need to rely on that contact you have with Adobe, if no one else chimes in here.  But perhaps in time someone else will see something you can consider or do.

I will say again that if there's ANY possible way you can create a test case (of just a few lines of code) that we can use to see if WE get the error, that would at least allow us to confirm if we experience the error (we may or may not). It would also dramatically help Adobe solve the problem, if they too are unable to recreate it based solely on the words you have offered.

Of course, your test case would need to somehow connect to something (that we can reach publicly) that uses what you're trying to use....so that we can confirm it "works" in the older CF update and then "fails" in the newer one/s.

I realize you may feel that's "impossible", if you think only that you "can't make your stuff web accessible". But I'm not proposing you do that. I'm proposing you find some way that WE are able to access SOMETHING that is web accessible which supports the connection approach you're attempting. I'm not saying it will be easy to find or arrange. I'm just saying that without it, we're stabbing in the dark.

Finally, FWIW, I have sometimes found that even for such seemingly very challenging problems, if I am able to get on a shared desktop consulting session with someone (perhaps even for just 15 minutes), I may help them find or assess something about the issue that was only possible to see "on their machine", while they remoted in and I "watched". I don't need direct access: we'd use Zoom or anything like it you may prefer. I know some people feel they "can't" do that, or they feel they "can't pay for help" (even just for 15 minutes).  Anyway, if you're at all interested, see the consulting page at carehart.org for more on my rates, approach, satisfaction guarantee, online calendar, email, phone, and more.

[moalspvic]

While similar, the FTP error is the simpler of the two to recreate...

It appears to be triggered during the private key processing before the actual connection attempt (hostname resolution or handshake). This issue is reproducible in both DEV and PROD environments.

• The system was working correctly prior to the update. No changes were made to the <cfftp> tag configuration.

• Since the error occurs in both DEV and PROD (despite PROD being whitelisted and DEV not), the issue seems independent of the host, credentials, or actual server connection. It appears that the error occurs when the system attempts to process the private key file.

Steps to reproduce

Create an RSA-SSH2 key pair.

Attempt connection using the private key file (.ppk):

<cfftp action="open"
    server="sftp.********.com"
    port="22"
    username="********"
    key="C:\********\privateKey.ppk"
    passphrase="********"
    secure="yes"
    connection="connFTP" />

(Note: The server, port, username, and passphrase fields can use any values, as the error occurs prior to establishing a connection.)

(Note: The system successfully accesses and opens the private key file. If an invalid file is used, the error is simply "invalid privatekey.")

There are a few nested cause structs, here are the cause messages from the inner and outer most

	Message
Inner Cause	org.bouncycastle.crypto.params.Argon2Parameters$Builder not found by jsch-0.2.17 [23]
Outer Cause	'argon2' is required, but it is not available.

[Charlie Arehart]

Well, rather than go about creating the key (and hoping to have it produce just the right variant), let's try a different diagnostic approach--something YOU can run on your end, to compare it to what I find on my end (in a stock CF2023 updated deployment).

So first, on your CF instance that's now running your code successfully (before you updated), run the following code. It will output the name of the file holding the java class of that type in the error message. (It could be written as tags, of course, but let's not worry about that now):

<cfscript>
class="org.bouncycastle.crypto.params.Argon2Parameters$Builder";
writeoutput(createobject("java",class).getClass().getProtectionDomain().getCodeSource().getLocation().toString());
</cfscript>

On my CF2023 (with update 17, which came out yesterday), it reports:

file:C:/ColdFusion2023/bundles/repo/bcprov-jdk18on-1.78.1.jar

Note the 1.78 version number there. That is above 1.61, which is when the jscf library in that jar added argon2 support (from my research. And to be clear, the 18 reference in the jdkon portion of the file name is not java 18--which CF2023 doesn't support. Instead it's how that project indicates that the jar is for use with Java 8 or above--which used to be known as 1.8, thus the "18".)

Yours may well be in a different folder (hold that thought), but does it produce any filename--again on the CF instance where your existing code does run. Let us know the full path, not just the file name.

Then second, note how that classname (from your error message) specifically refers to argon2, and my code here should only work if the argon2 class is available to CF. As such, I expect it would fail on the CF instance where your code gets the error you report.

Do go ahead and run it there also, and report what it returns (even if an error).

We may find that BEFORE you updated, CF somehow DID find a java jar (or class) file that holds that updated jsch library. But then when you updated, somehow that was lost. Again, my CF2023 update 17 DOES support it, judging by the info above.

[moalspvic]

Charlie, Interesting approach.  To note, I do not currenlty have a machine that this is working on.  It worked previously on the PROD machine, however is no longer working since the update.  I've brought the DEV machine into the mix for testing, however the update was already applied and currently exihbits the same behavior.

I will run through your troubleshooting steps and let you know what is reported.  

Thanks.

[Charlie Arehart]

Fair enough, moalspvic. But I assume you mean first that prod has been updated and for now the update has not been uninstalled. (Perhaps pressure will change in time for you to do that, if this remains unresolved.)

In the meantime, since it worked in dev BEFORE the update and doesn't AFTER, can you uninstall the update on DEV?

If you may try it (on either instance), please do run that test code I offered, first BEFORE uninstalling the update, to see what it reports is serving up that argon2 support for you now. (You may even find it's some jar other than one Adobe has provided.)

Then for sure after uninstalling the update run it again. It might reports some still OTHER jar which was in place before the update but was removed or changed by the update (whether it was one Adobe provided or on your folks put in place).

People often do that with regard to bouncycastle and its bcp*.jar files. They often feel they "must", to solve some problem. But then such manual tweaks might "work" for one CF version/update level but then NOT with another. I'm simply trying to help you determine if this may have happened for you.

Otherwise, others are trying to help and perhaps you may find resolution/confirmation another way. As always, just trying to help.

[moalspvic]

Charlie,

What I mean is that PROD and DEV have both been updated.  These were sections of code that I did not have on my DEV, so they were not tested prior to updating PROD.  Unfortunately, as this time, these processes are down in PROD.  However, I did find a VM snaphot of DEV prior to the update and ran your snippet, as well as the CFFTP call on it.  The results follow:

PROD and DEV: your snippet throws an error 

ClassName	org.bouncycastle.crypto.params.Argon2Parameters$Builder
Detail	Class not found: org.bouncycastle.crypto.params.Argon2Parameters$Builder
ErrorMessage	org.bouncycastle.crypto.params.Argon2Parameters$Builder
Message	Object Instantiation Exception.

DEV VM: file:C:/ColdFusion2023/bundles/repo/bcprov-jdk18on-1.78.1.jar

I looked for that file on DEV and PROD and it is on both.

A bit of additional background regarding intial trials going forward and backwards on DEV

Prior to reaching out for help, after the initial error was discovered in PROD, I noticed that Update 17 was released and contained the CFMAIL patch (which was seemed similar), so I applied Update 17 to DEV and it failed again. At that point, I rolled back to Update 14, using the uninstaller.jar on DEV and it continued to fail...which is why I hadn't rolled PROD back yet.  It seemed like the problem wouldn't be solved by simply running the uninstaller.jar. I then turned to Adobe Support and The Community for help.  Adobe has been somewhat non-responsive.  The provided a FTP jar to replace an existing one in the bundles folder, but that didn't work either.

Now that we know the DEV VM works and DEV and PROD do not, what are your thoughts on a way forward?

I appreciate your help sir. Thanks!

[BKBK · Accepted solution]

Hi @moalspvic , it appears as if your application fails to load one or more packages related to encryption. So I would suggest that you install all the packages. At least, to start with. 

Assuming you are on Windows, you could proceed as follows:

1. Stop ColdFusion 2023;
2. Open the Command Prompt (cmd) as Administrator;
3. Use the DOS cd command to navigate to  {CF2023_HOME_DIR}/cfusion/bin;
4.  Type cfpm and press ENTER. You should get ColdFusion's package manager prompt cfpm;
5.  Run the command uninstall all
   
   
6. Wait for the CFPM tool to finish uninstalling all the packages.
   Then, leaving the Command Prompt window open, restart the ColdFusion service;
7.  Run the CFPM command install all

8.  Note down the name of any package that the CFPM tool says has not been installed or could not be installed.
Keep running the command install all till the result is either an irreversible error or ColdFusion tells you that "All the packages are already installed".

9. If any packages were not installed or if there were errors, then share that with the forum. Otherwise, restart the ColdFusion 2023 service.
10. Test by running your cfftp code.

[Charlie Arehart]

That's an interesting theory, bkbk. But in that case wouldn't the simpler question have been simply to ask moalspvic to report what packages, if any, were listed as not installed (such as are listed in the CF Admin display of "available packages")?

And if they did alredy have all installed, then what really could be expected by doing such a complete uninstall all then install all? To be clear, they'd already reported that "All packages were downloaded successfully" and they confirmed seeing in the logs "a long list of packages being unistalled, then a list of packages starting...without error".

Please don't get me wrong: I'm not saying someone "shouldn't do what you pose" or that you were incorrect to propose it. But I am saying that it seems a bit of using a hammer where a screwdriver might have been sufficient. If nothing else, I would want to propose this question (what packages are not installed, if any) before they bothered with all that effort. (And FWIW, doing that "install all" may install packages that they didn't WANT to install and that may have nothing to do with solving the problem, so it's a potential side-effect folks should be wary of when it's suggested.)

No need to repeat your mantra that this is an open forum and that there are many paths to an answer. I do truly understand it. Consider this more of iron sharpening iron. In any case, I do certainly look forward as well to whatever moalspvic may have to offer in reply to my earlier suggestions in the other thread here. Inded, their running that simpler diagnostic and reporting the finding may be a lot less effort (and have no side-effect).

[moalspvic]

BKBK, thanks for reaching out.

The PROD machine in question has all packages installed already.  (I know, not best practice) and as noted in the previous message with Charlie, there were no errors noticed in the logs during the application of Update 15/16 and the subsequent startup of each instance.  Additionally, I am using a DEV system (as I cannot test and we want to make minimal changes to PROD) to test this.  It has all but search, spreadsheet, and websocket installed. This machine was updated first and likewise, there were no errors noted in the logs during the application of the update and the subsequent startup.  Both machines, at this time, are showing the same error.

I can perform your troubleshooting steps and let you know what is seen, however thought it important to share our current status.

Thanks again.

[BKBK]

BKBK, thanks for reaching out.

The PROD machine in question has all packages installed already. 

By @moalspvic

I think you misunderstand. What I propose is that you re-install all the packages using the CFPM tool. The hope is that the process will download and install any missing packages.

[moalspvic]

This approach worked on DEV. Although there were no errors during the update, unistall, and subsequent install of all updated packages... something was corrupt and, at least for us, was duplicatable. Performing a complete removal, download of packages and dependcies, and reinstallation cleared both issues that we were experiencing.  Now, to Charlie's point, what was the underlying issue... I don't know.  However, because this client has all packages installed already, using the hammer as a screwdriver worked.  I will do this on PROD this evening and report back. 

Thank you.

[BKBK]

Thanks for the update, @moalspvic . Fingers crossed - good luck!

[moalspvic]

I performed this on PROD last night and it the two service that were failing are back in operation. I appreciate everyone's help with this. 
Thank you. 

[Dave Watts]

I'm not putting a lot of thought into this, but if you can, just regenerate new keys for your users without using the argon2 encryption library.

[moalspvic]

Dave, thanks for your thoughts.
As requested by the third party provider, we created RSA SHA2 keys.  We did not specify the algorithm.  We have a key on PROD and I created a key for my DEV machine. Both were created using puttyGen.  So to say, create new keys without argon2, we didn't specifically request argon2. Additionally, as similar issue, with different error (noted above) is taking place when trying to authenticate using a different key for a webPush service. We're just starting with troubleshooting the ftp issue first as it is simpler to reproduce. 
Again, noting that both of these prodecures worked prior to the application of Update15/16 and associated package updates. 
Thanks. 

[Dave Watts]

My thought was that Argon2 is baked into openssl, and if you couldn't get CF to support it, you could (ugh) downgrade openssl to a prior version that still uses SHA-256 and regenerate users' keys. But then I looked it up, and the last version of openssl that uses SHA-256 by default is 3.1 - which is pretty old. The Argon2 encryption library is, I think, more suited to end-user PKE than SHA-256, so if you can add Argon2 support by switching CF or Java libraries, that's a better solution than mine.