ColdFusion 2021 Update 11 New Connector Required

[neowire]

Requesting some help with this one.

I know the basic steps for creating the new connector. I am able to do that successfully, but upon using the new isapi_redirect.dll the isapi_redirect.log is being written with this error upon trying to access the ColdFusion Administrator: [warn] jk_check_path::jk_util.c (2476): Blocking this uri: [/CFIDE/Administrator/index.cfm] since its starting with cfide

I know I've seen numerous posts about not accessing the CFIDE and that Adobe is helping us by blocking it, but this is my only way to administer this in my environment.

If I go back to using the previous isapi_redirect.dll there is no issue and I am able to access the ColdFusion Administrator console without any issues.

Is there a new exclusion that we need to include somewhere to allow access to CFIDE/Administrator with this new update?

[Charlie Arehart]

Have you added the ip address from which you want to connect, in the "allowed ip addreses" page of the CF Admin (in the Security button on the bottom left), and specifically in the second box of that page? The first is unrelated to protecting the CF Admin itself.

FWIW, the technote for the CF2023 update did refer to a change that indicated a change in new installations regarding how IPs could be configured in the installer. From folks like you and others, it seems it's MORE than just new installs of CF (there were new installers made available), but also merely in upgrading the connector (that isapi_redirect.dll, in the case of IIS).

And it also is clearly affecting CF2021, though the technote has no mention of it. See a discussion of that in one of the comments on the original post about the updates from last Friday, by Brian, here.

It would be helpful to get clarification from Adobe about that, as there was no mention of 

[neowire]

@Charlie Arehart 

Thank you for your response. Our set up for ColdFusion 2021 in our environment is locked down pretty much in accordance with the out of date STIG for Adobe ColdFusion 11. Yes, we have the IP address necessary to have access added (rather a wildcard range) and that has been working successfully. And it continues to work successfully, until the isapi dll file is replaced.

[Dave Watts]

I'm not too familar with the CF 11 STIG. Is it possible for you to avoid using IIS for accessing the CF Administrator and using the built-in web server for that? You should be able to run them both, limit access to the built-in web server via IP addresses in Tomcat instead of CF, and only allow access to the CF Administrator via the built-in web server.

Dave Watts, Eidolon LLC

[neowire]

Good recommendation, but we are required to disable the built-in web server and are instead using IIS.

[Dave Watts]

Are you able to set up an additional IIS virtual server, and use that just for administration? You could put your access control constraints in that instead.

Dave Watts, Eidolon LLC

[Brian__]

@neowire and Charlie - I've observed the same the new connector behavior you both mention. The release notes don't say much about the new connectors, but they do state "Wsconfig update now creates a backup of the modified files." I haven't gone through a ColdFusion upgrade to either new release, but I can say that running a "wsocnifg -update ..." on a new CF2021U11 server will back up the connector config files and library. So reverting to these backed-up files might be the best way to revert if needed. (@neowire it sounds like you went that route in restoring the previous DLL...)

However - for those that don't need remote access through a connector to CFAdmin, this is a considerable security improvement. I've shared some more thoughts on that here - https://hoyahaxa.blogspot.com/2023/10/coldfusion-connectors-and-cfadmin.html

[cat_zilla13]

@Brian__  Very nice summary of connectors.  However, you discuss "if you don't need remote access to CF Admin", but leave out what if you _do_ need remote access?  We run on a hosted IAAS Linux platform with Apache web server.  I can't run CF Admin on localhost.  We have used an external web server to host CF Admin for longer than I've been on the project, making use of the Apache Tomcat AJP Connector.  We have a separate URL for CF Admin with SSL certificate, and employ firewall/proxy rules and allowed IP addresses to restrict access.

While it is true that wsconfig now creates backups of the modified files when creating a new connector, doesn't reverting to an older connector sort of defeat the purpose of upgrading Tomcat?  

You state that URIs beginning with /CFIDE are blocked in the new connector.  Is that inherently built in to the new library?  Or is it a config setting somewhere that can be disabled?  Is there any recourse for those of us who need CF Admin somewhere but are discouraged from running an outdated Tomcat due to security concerns?

[Brian__]

@cat_zilla13 Thanks and those are all great questions.  Maybe someone from Adobe will chime in with definitive answers versus what I've been able to figure out by feeling around in the dark.

> While it is true that wsconfig now creates backups of the modified files when creating a new connector,

> doesn't reverting to an older connector sort of defeat the purpose of upgrading Tomcat?  

I believe that you should be able to upgrade ColdFusion (which will upgarde Tomcat) and just not re-create the connector.  Or revert back to the old connector library and config if you do re-create it and then want to go back.  AFAIK, this won't revert to the older version of Tomcat.  (You won't get the added security of the new connector, but it also won't block /CFIDE access.)

> You state that URIs beginning with /CFIDE are blocked in the new connector.  Is that inherently built in

> to the new library?  Or is it a config setting somewhere that can be disabled? 

Yes, this is compiled into the newest versions of mod_jk.so and isapi_redirect.dll.  I'm unaware of any ways to bypass these checks built into the new connectors.

[cat_zilla13]

My thanks once again to @Charlie Arehart and @Brian__  for the guidance.  Long story short, I've got CF Admin back on Tomcat and SSL/TLS-enabled.  Once again, I would like to point out the lack of documentation for importing existing certificate and CA chain into the Tomcat trust store.  Lockdown guide only covers self-signed certs, which doesn't meet security standards.

[dejank52055655]

@cat_zilla13Do you by a chance have instruction on how to get the CFadmin on Tomcat?

[cat_zilla13]

Without knowing anything about your environment, I can't give specific instructions.  I basically went back to the ColdFusion installation and lockdown guides as reference and tried to restore original config since ColdFusion installs CF Admin on Tomcat by default. 

I ended up having to open a port on our server for Tomcat/CF Admin to separate it from our external webserver.  All references to the CF Admin URL were removed from the external webserver configuration and I uncommented the Connector tag in $CFUSION_HOME/runtime/conf/server.xml.  Since we require secure HTTPS connection to CF Admin, I had to create a Java truststore and import the server certificate, then updated the Connector in server.xml to point at the truststore.

[dejank52055655]

Thanks, I was just looking for generic description of what you had to do to get it working, thank you I appreciate it.

Our enviroment is locked down, in private network behind firewalls and reverse proxy, no access to CF admin portal through external webservers at all.

I'm not quite sure why Adobe did not provide config that can be used to make simple exception, they are making it pretty diffcult with the workarounds.

[Charlie Arehart]

I want to expand on neowire's comment above about this issue being a conflict with the cf stigs (DOD/NIST standards.)

Put simply, this particular STIG requiring disabling of Cf's built-in web server is a indication of how dated the cf stigs are (as many have long complained). Don't stop reading if you disagree or feel you have no choice in following them though. 🙂 After some historical perspective I offer more suggestions and thoughts on the issue above. 

1) First, as some here may know, since cf2016 Adobe always ENABLES  that built-in web server on install of cf. Only in cf11 and before was this an option. And they did this SPECIFICALLY for using it to access the cf admin. 

(And FWIW, the built-in web server is not the old, poorly supported jrun web server it was in cf9 and earlier, which is when Adobe and Macromedia themselves recommended against enabling for anything but development. Instead since cf10--2012--it's the Tomcat web server, which many orgs even use for production, if not "needing" the additional capabilities of a "real" one fronting it, like iis, apache, nginx, etc.) 

At the same time Cf2016 intentionally enabled that built-in web server by default, it also DISABLED accessing of the cf admin via an external web server, by their adding (for us) the line to block urls with CFIDE in the connector's uriworkermap.properties file. And folks following the STIG felt compelled to remove that because they "had to enable accessing the cf admin via other than the web server".

But doing that opened them to the very vulnerabilities which Adobe had long been trying to protect folks from, in closing down access to that CFIDE folder on other than the built-in web server. (For years, the lockdown guide had recommended people do that via configuration of their web server, but this connector change instead enforced the behavior.) 

And this new change (in the connector of the most recent update) simply takes things farther, for the sake of our security, building that block of CFIDE in a way that does NOT rely on that uriworkermap.properties block, which not only could be removed but could be circumvented even if in place (it was case-sensitive, for instance). 

2) It's very much in the interest of most folks to use the built-in web server for accessing the admin, since it uses a non-standard port (8500 by default) which would be blocked by any firewall. Then THAT firewall hole could be opened to only authorized users. (And it need not be run on localhost only.) 

But are there things that an external web server (iis apache, nginx, etc) can do which may not be supported in the cf/tomcat web server? Sure. Still, it's also far more configurable than most realize. See the Tomcat docs and other resources for more on that. 

In any case, another option is to use an external web server but set it up to proxy/port-forward to the built-in web server, which avoids using this ajp connector which Adobe has modified to now block CFIDE access more completely. (Of course, it then puts the responsibility back on such folks to thoroughly protect that admin access in that web server, which Adobe was trying to enforce for the majority of people who just "install and go".) 

Or use ssh tunneling, discussed even in the latest lockdown guide still in its section 6.4.

3) All this is a reflection on how dated the stigs are, literally with cf11 in the title and url:

https://www.stigviewer.com/stig/adobe_coldfusion_11/

Still, I know that folks who "have to follow the stigs" feel they have no choice, and that they have no say in getting them improved. This may be one of those matters that forces the issue to be confronted.

4) FWIW, the stig entry on this point, "The ColdFusion built-in TomCat Web Server must be disabled", is listed as a medium rather than high priority. Do folks facing the above dilemma really do ALL the mediums, including the two on sandboxing? I'm willing to bet most do not.

5) BTW, I have had no say personally in all this (the cf features, the recent changes, the stigs). I'm just an observer and messenger, as I help folks in my day to day work, as well as here and elsewhere.  I'm not arguing but just presenting things for consideration. I'm open to feedback, of course. And Dave's shared helpful thoughts folks can consider as well. 

And I only discerned the connector change yesterday. I'll  have more to say in a blog post. Until I may do that, here's a bit more. 

6) First, you can SEE the requests being blocked (this new way, after updating the connector) in the log file that's long been present in the connector folder (under cf's config/wsconfig folder, in the numbered subfolder for each connector, and where the file name for the log varies based on whether using iis or apache).

And maybe there's more to the new behavior than we've been told. Maybe there's a way to cause a variation on its default blocking. Time will tell. The connector dll (for iis) or so file (for apache) is binary and we're not given the source code. 

Finally, none of this was mentioned in the update technotes, which is indeed frustrating.  Again it's only coming out as people proceed with the update of CF--and then if they also remember to update the connector. We'll likely see the "discovery" repeated over and over in months and years to come, as people move to the updates or indeed to cf2023 and 2021.

That's all the more reason I've shared all this here. And I'll likely create a blog post based on it (this stig aspect, I mean), as well as one on the connector change. But I'd like to hear the feedback here first--for those who had the stomach to read all this. 🙂 

[cat_zilla13]

@Charlie Arehart 

I am in the same situation as @neowire in that I am expected to set up our CF 2021 environment in accordance with the outdated CF11 STIG where other documentation is not available to provide as evidence.  I have been able to point to the Adobe ColdFusion 2021 Lockdown Guide 1.1 for some updated recommendations, but there is no specific documentation from Adobe that I can find that indicates that the built-in web server is now preferable to using IIS/Apache for accessing CF Admin, and that the original reason for disabling Tomcat for CF Admin is no longer applicable. 

For reference, the discussion text in the STIG:

Application servers provide a myriad of differing processes, features, and functionalities. Some of these processes may be deemed to be unnecessary or too unsecure to run on a production DoD system. The built-in TomCat Web Server is used to host the Administrator Console and is used for initial setup. While the built-in server can be used to continually host the Administrator Console, this is not the best practice since the server is not guaranteed to be patched and upgraded, implementing TLS is not well documented, allowing for poor implementations, and commercial web servers offer better logging. To enable the Administrator Console to still operate and disable the built-in TomCat Web Server, the Administrator Console application must be moved to the web server (i.e., IIS, Apache, IBM HTTP Server, etc.) hosting the ColdFusion applications. Moving the Administrator Console to Apache and IIS is well documented in the Adobe ColdFusion Lockdown Guide.

If there is documented evidence from Adobe that indicates built-in server is the way to go, point me at it! However, without said evidence, I'm stuck with disabling Tomcat for CF Admin, which apparently doesn't work for CF 2021 update 11.  Open to suggestions.

[Charlie Arehart]

I've got nothing to point to, other than the fact that Adobe HAS both enabled that built-in web server by default and AND blocked admin access via iis or apache since cf2016--now 7 years ago, and they have not changed that since.

Indeed, they've doubled down with this change. If that's not evidence enough to seek exemption from the  stig, I don't know what will be.

Again, do you setup the sandbox feature in cf? It's a stig requirement of the same priority.

As for the inability to update tomcat, that's a serious issue. Nothing we can do about that but wait for updates. Press Adobe about that. Again, I have no say at all in such matters. Same with them improving the configurability of the built-in web server, such as from within the cf admin. 

[Charlie Arehart]

Btw, there's more evidence of the dated nature of that discussion:

"Moving the Administrator Console to Apache and IIS is well documented in the Adobe ColdFusion Lockdown Guide" 

Um, no, it is not. Not a word about that since the cf2016 guide which last covered it in its section 2.23 for iis and 5.8 for apache. 

[cat_zilla13]

>"I've got nothing to point to, other than the fact that Adobe HAS both enabled that built-in web

>server by default and AND blocked admin access via iis or apache since cf2016--now 7 years ago,

>and they have not changed that since."

Respectfully, that seems inaccurate as my organization has been following the STIG guideline for several years to disable CF Admin on built-in server and configure to run on external Apache.  This has been done in CF 2016, CF 2018 and now CF 2021.  It has been working without issue right up until update 11.  Specifically, it was recreating the Connector which broke it, not the actual patch.  I don't suppose you know _how_ CF Admin is being blocked?

>"While the built-in server can be used to continually host the Administrator Console, this is not the

>best practice since the server is not guaranteed to be patched and upgraded". 

This is the crux of the argument used by my origanization's security team for running CF Admin on external web server.  Even with the latest update, Tomcat is only now patched up to 9.0.78 which leaves several potentially open vulnerabilities which have been addressed in the current version 9.0.82.

>"Moving the Administrator Console to Apache and IIS is well documented in the Adobe ColdFusion

>Lockdown Guide" 

>Um, no, it is not. Not a word about that since the cf2016 guide which last covered it in its section

>2.23 for iis and 5.8 for apache. "

I certainly can't argue with that.  

[cat_zilla13]

Sorry, in answer to an earlier question, yes we use the sandbox feature.

[svnfightsvn]

We ran into this exact same issue after installing CF2023.  We previously were running CF2018 and skipped CF2021.

For CF2018, we followed the lockdown guide to disable the built-in Tomcat webserver (comment out the port 8500 connector) and in order to access CFIDE, we added a line to the uriworkermap.properties to get access to it through IIS.  Of course, this no longer works in CF2023 because it seems that the isapi_redirect.dll is the same that CF2021 update 11 installed.

This is pretty annoying I must say and I'm not a fan of how this was implemented without any warning.  In our environment, our IIS server is already behind a very restrictive firewall and access to anyone outside of our admin subnet happens via a reverse proxy that prevent access to CFIDE (in addition to the IP address restriction in the admin console). 

With this new change to the isapi_redirect.dll connector, the ONLY way to access CFIDE is through Tomcat.  On top of that, you have extra work to do in order to secure that access over SSL by manually building a keystore, changing server.xml, etc.  Plus, in order for that certificate to be trusted by browsers, it must be signed by a known CA and needs to be updated every few years.  What a pain.

The steps we took in CF2018 provided us with the security we needed/wanted (our cyber team scans these machines religiously) and it would be helpful if Adobe reverted this or provided some way to allow it (at your own risk).

@neowireif you want to go back to the "old way", its possible to use the old isapi_redirect.dll (I tested it and it works).  Of course I'm sure this is frowned upon by Adobe and updates may revert it back...

[Dave Watts]

Maybe Adobe isn't great at notifying customers of the product roadmap, but you can participate in the prerelease cycle and find out what's being changed. As for SSL/TLS, you can use the same certificate and private key that you use for the web server. That's really all that needs to be installed in Tomcat. The installation process is a bit more clunky, though. But you're already updating your web server certificate every three years, right? You can probably set up Let's Encrypt to update it even more frequently.

https://www.revolgy.com/insights/blog/letsencrypt-tomcat-on-windows

As for the change itself, I sympathize with you. But the new arrangement may well be more secure, even if it doesn't conform to the CF 11 STIG. What are our security goals for CF admin access? Ideally, it should use a different channel from public web access, so that you can use a separate set of access control processes. In other words, if I have CF Administrator available through Tomcat on a different port, I can use a different firewall rule or security group or whatever, or even limit access to localhost and use RDP to connect to the server then use a browser on the server console to connect to the CF Administrator. Those are pretty simple examples. Maybe I do create a separate certificate that's signed by an internal CA instead of a public one, and require internal browsers to support that CA for management purposes. Etc, etc.

Finally, as for cyber teams scanning machines religiously, I dunno about that. I mean, lots of contractors use Nessus but don't really know any mitigation strategies other than "patch to latest version". That's not going to protect you against 0-days.

Anyway, I realize that this could all be read as a criticism of your statement. It isn't. You shouldn't have to do it. But you do have to, and you can.

Dave Watts, Eidolon LLC

[svnfightsvn]

Your right, we shouldn't have to.  I guess thats my gripe....we no longer have a choice.  I understand having this setup as the default, but at least give organizations the ability to make it less restrictive if they choose.

Also - we cant use the same SSL setup as the web server...its IIS.  There isn't an easy way to make Tomcat use the same private key and certificate as the IIS web server.  And renewing an IIS certificate is just a few clicks in a properly setup Windows domain.  Its a bit more complicated on Tomcat but still not too difficult...we run home grown Java apps on Tomcat so I'm very familiar with the process.

We already have a VERY restrictive firewall setup for direct access to these CF servers and we also dont want to have to RDP into the machine just to access the CF admin console.  We dont have to get into it very often, but its just another hurdle thats a PITA when you do need to. 

[TheNephalim]

I am in full concurrence with your assessment about making it the default, but allowing administrators to opt out of it.  

We are currently in the process of upgrading from 2018 to 2023.   We have our ColdFusion Administrator setup as a separate website with an internal IP address that's not publicly accessible, uses SSL/TLS, and configured to deny anonymous users.  It's locked down according the ColdFusion and the IIS STIGs.  

As I mentioned to Mr. Arehart on another thread, this really puts us in a difficult position because, from what I recall, we cannot access browsers directly on our server.  To access the administrator, we would have to put in HBSS exceptions to administrate ColdFusion which will present some issues I'm sure.

[Charlie Arehart]

As far as calling for a change in the behavior, I'll note that Adobe has never responded here. While there's value in the public discussion that's been had here (especially for those open to considering alternative solutions), those wanting to see this CHANGE should file a ticket at tracker.adobe.com. 

Whether you may file it as a feature request (to add more control over this change) or a bug report (to assert Adobe was wrong to "just change the behavior out of the blue" in Oct 2023), please do report the link here, so interested folks can add their votes and comments there.

It would be wise to point also to this thread when creating the ticket, so readers there know about all that was discussed here.

PS I believe you're the first person in my dozens of years here to refer to me as "Mr Arehart". Please, do call me Charlie. 🙂 

[TheNephalim]

I really want to thank you for the response.  

I plan on following the suggestion of a bug report. 

I really cannot adequately express my frustration that they implemented this without providing the option of opting out.  It's even worse that they didn't document they were making this change.  I reviewed the documentation again for Update 5 to see if I missed something.  I didn't see it and if it's there, it's really buried in there.

Sorry about the "Mr. Arehart."  My father drilled into us that you didn't call people by their first name unless they gave you permission.  I think it was our Pennsylvania German upbringing.

I also really wanted to thank you again for your assistance.  For a variety of reasons, I'm late to the ColdFusion 2023 party and so probably dredging things up again when people thought it was dead.

When I file the ticket, I will definitely post the link here.