ColdFusion 2018 - HTTP Status 403 - Forbidden

[HC_Tech]

Hello,

We have ColdFusion 2018 (HotFix4) running on (Windows Server 2016 - IIS).  The ColdFusion Lockdown has been installed.  The site was working fine with HTML & CFM pages, until something changed. Now only HTML pages work.  When we attempt to browse to a CFM page, we receive:

HTTP Status 403 - Forbidden

Description The server understood the request but refues to authorize it.

Apache Tomcat/9.0.10

I confirmed the Application Pool identity account has access (Read/Execute) to the CFM files.  There is an entry under "ISAPI Filters" named "tomcat' that points to (D:\CF_Install_Folder\config\wsconfig\1\isapi_redirect.dll), and this file exists.  The (wsconfig.exe) runs fine.

Does anyone have any idea what may be causing this issue (403 - Forbidden) for CFM pages?

Thank you!

[Charlie Arehart]

Did you ensure that app pool identity user has access to the config/wsconfig folder you name? It needs write access for the logs it would write. And while you could do this only at the /1 folder, you may was well do it at the config/wsconfig folder above it, in case you may add still other connectors later.

(Yes, if each site has its own app pool that uses only its own connector/numbered folder, then yes, you need only give THAT app pool identify user permission to that specific folder, if you want to be really secure about things. But really, this is not about getting access to CF code or CF settings, but just connector config files, so it may not be that critical to be so careful. I am open to correction from anyone interested.)

Let us know if that change helps. BTW, you mention that applied lockdown. Did you do that just before things started to fail? Or weeks/months ago? Did you do it by hand? or by using the auto-lockdown tool new to CF2018?

[HC_Tech]

Hello Charlie.

Thanks for pointing me to the (config/wsconfig/1) folder.  We did have some incorrect permissions, which I have corrected. Fortunately, we have two CF nodes configured identically.  I mirrored the permissions between the two nodes. The second node still works fine, and I'm able compare settings between good and bad nodes.

Unfortunately, I am still receiving the 403 error.  We have three IIS web sites on this server and all report the same error.  They all appear to be using the (/1) folder.

During the initial build I installed the CF auto-lockdown tool (.exe) on both systems, and they continued to work fine. This was several weeks ago.  If the file permissions have changed, I have not yet been able to identify where?

[Charlie Arehart]

Have you tried making the request to the "failing sites" from the server itself, on which they run (rather than from off the server)? You may get a different and more useful error message. Assuming you still get just the 403 and no more, check the "error pages" feature in IIS for the site you are testing. Use its "edit feature settings" on the right. Is it set to show details for local and custom errors to remote users? If it is instead "custom errors" for both, switch that for a moment to detailed for local (not for remote), and try the request again from the server.

If that doesn't help (or you can't or won't be able to change it), another thing you can try if the IIS Failed Request Tracing (or FRT) feature, to find more on why the requests are failing. That's too much to cover in a simple reply here, but google searching will help, or I can help via remote consulting (perhaps having all that done in minutes).

[HC_Tech]

I am familiar with IIS detailed error pages and tried that.  It only reported the same 403 error both on the server and remotely.

After fully comparing and adjusting the file permissions as needed, between the working and failing node, I was still unable to resolve the issue.  I ended up performing a "clean" CF re-install (CF 2018, Patch, and Lockdown), which resolved the issue.  It is unfortunate, I had to take such a drastic step, as I now do not know the root cause.  I have captured VM snapshots as well.  Need to setup VM backups next.

Thanks for all of your help and advice Charlie.

[Charlie Arehart]

Glad to have helped. Since you have the VM, if you wanted to know what was up (and how you may have solved it without doing the unistall/reinstall), I probably could help do that. If I didn't help, you wouldn't need to pay for the time. But I understand that you may just want to leave it at a curiosity solved by the reinstall.

[berniemiller]

I had the same issue after running the Lockdown tool. I ran wsconfig.exe manually and tried the Upgrade button. I have no idea what that does but it fixed the issue for me.

[arshad_brody]

Thank You!

I also received the 403 error after I installed the ColdFusion 2018 Performance Monitoring Toolset.

[Dani Szwarc]

This has fixed it! Thank you Bernie for posting it!

[Charlie Arehart]

Dani, I'll share that as for fixing 403 errors by running the wsconfig tool to use its upgrade feature, that's somethiing needed after applying the update for CF2018 (and 2016) that came out in March 2020. I know you say you applied the auto-lockdown tool, and this thread (started in Oct 2019) was about an issue related to that.

Just clarifying in response to your observation. And for the sake of others who may find this, I'll note that for more on the 403 issue (and still other solutions to it, related to the March update), see my post from that time (https://coldfusion.adobe.com/2020/03/three-reasons-sites-may-break-fix-applying-mar-2020-update-cf20...), which also points to Adobe update docs and yet another blog post of mine with still more detail for those who may want it.

[davidr37592242]

With "ease of use" as one of the current marketing focuses, please relay to the team that such breakages should be supremely advertised during hot-fix installation, after installation, in the installation logs, on the download page. Optimally the hotfixes would run every step required to bring the ColdFusion installation being fixed into a valid state. Users should not be expected to search forum posts to discover why every HTTP request on their machine is now serving a 403.

As developers we understand that breakages like this occur, but the workflow for addressing them here needs improvement.

[Charlie Arehart]

David, I hear your frustration, but while I don't work for Adobe I do think they have done more than most folks realize. 

This potential issue is the first thing mentioned at the top of the technnote for update 10. (The problem started with update 8, and it points out that if one is updating from anything before 8, they need to follow the "post installation" steps indicated.)

And every update (in the update UI) offers a "read more" link to the technote for that update. Sadly, most folks don't bother. And since the updates are indeed cumulative, they can't list in that update UI box ALL the things that may have happened in the updates one may be skipping over.

FWIW, each update technote does offer a link to a page with all the previous updates and their release notes. Does it suck that someone may have to review those? Yes, but again note how the top over every technotes identifies key issues like that one, as well as the one from Update 4. 

As for wishing the update mechanism would do "all the steps needed", there has always been a separation between the updater (which updates CF) and the web server connector (which runs as a separate app, the wsconfig.exe). And FWIW, every update technote indicates whether an update to the connector is required for that update, and since CF11, every technote lists at the bottom what prior updates required such a connector update, which is really helpful again when one may be skipping updates.

So again, all I'm saying is that they have done more than most may realize. Could they do more? Sure. And again, I appreciate the frustration you're expressing and that it's shared by many others.

I will say finally that this forum may not be the place to plead for that. Instead, file a feature request (or bug report) at tracker.adobe.com. That's what Adobe watches for "things to fix". We rarely see them comment on anything here.

[T27332149bvar]

My similar-problem was with Coldfusion2021 & Apache24 (64bit) on fresh Win11 i9 laptop.

Note: "C:\ColdFusion2021\cfusion\runtime\bin\wsconfig.exe" wasn't working. It would exit immediately with no message.

Problem 1: Connection unavailable
Solution:

Changed localhost to 127.0.0.1 in three places:
1,2. In "C:\ColdFusion2021\cfusion\runtime\conf\server.xml":
<Engine defaultHost="127.0.0.1" name="Catalina" jvmRoute="cfusion">
<Host autoDeploy="false" appBase="webapps" name="127.0.0.1" unpackWARs="false">

3. In "C:\ColdFusion2021\config\wsconfig\1\workers.properties":

worker.cfusion.host=127.0.0.1

Once this was fixed I had the 403 error.

Problem 2: Status 403 Forbidden
Solution:
There was a missing worker.cfusion.secret setting in "C:\ColdFusion2021\config\wsconfig\1\workers.properties" i.e.
worker.cfusion.secret=f3ae2c16-bd42-4ea1-88a9-[etc]

...but that secret key was already placed
In "C:\ColdFusion2021\cfusion\runtime\conf\server.xml"
<Connector protocol="AJP/1.3" port="8020" redirectPort="8453" secret="f3ae2c16-bd42-4ea1-88a9-[etc]" maxThreads="500" connectionTimeout="60000" tomcatAuthentication="false" />

Once I set it everything worked again.

I don't know if this will work for anyone else but I tried everything I could find in the forums before succeeding.

[William31143467d2v6]

You have no idea how helpful this community has been for me! Same exact error Coldfusion 2018 Update 17. The wsconfig worker.cfusion.secret property/value was missing after the install. Either manually adding the 'secret' from the server.xml file or following Charlie's method in the link below (probably the better method), problem fixed. Thanks!

https://coldfusion.adobe.com/2020/03/three-reasons-sites-may-break-fix-applying-mar-2020-update-cf20...