CF2018 - second instance responds with Tomcat 403 Forbidden status on any CF resource

[FabianCF]

Greetings everybody,
we are currently running a Windows Server 2016 server with IIS. We have installed ColdFusion 2018 and successfully run one IIS website (let's call it "does.work.com") connected to the default "cfusion" instance.
The server has one IP.

Now we added a second instance (let's call it "dnw") via the Enterprise Manager in the CF administrator. Afterwards we added a second website to IIS. Let's call this website does-not.work.com. Since we only have one IP we're using two different host names and checked "Require Server Name Indication" for both websites. 

does-not.work.com outputs HTML correctly.

After checking this we ran the Web Server Configuration Tool to connect the "does-not.work.com" IIS website to the second CF instance "dnw".
However (also after re-booting the server, re-starting IIS, re-starting the CF Application servers, ...) we receive the following error when calling any ColdFusion resources on does-not.work.com:

At the beginning we looked at missing permissions and also ran the Lockdown Wizard to lockdown the "dnw" instance. However we cannot seem to get it to work.
Unfortunately the error description is in German but roughly translated it means "The server understood the request, however denied authorization."

Can you please advise anything to help us to get this second website/second CF instance working?
Your help is greatly appreciated - thank you.

[Priyank Shrivastava.]

Hi,

if you have created the connector with all and then created the second instance. First, try removing both the connectors for instance1 and instance2. 

Create the connector with instance 1 to Site1 and the same for site 2 with instance 2 and see if that helps.

[FabianCF]

Thank you for your reply.
We actually haven't added the connector with all but I gave it a shot anyway.

For instance 1 now again connected to Site1 it is working as expected. For site2 with instance 2 it is still not working with the same error message.

[Charlie Arehart]

In addition to what Priyank shared, I will add that you don't indicate if you had run the CF2018 update mechanism, after initially installing it. Did you? And if so, did you do that before or after you added the new instance? If you did it AFTER adding the new instance, did you tell the update mechanism to ALSO update the new instance?

This is important, because the 403 error MAY be related to an issue that is related to update 8 (or above). And if you may have run the wsconfig tool AFTER the update but BEFORE updating your new instance, you would be running the wsconfig tool that WAS updated against an instance that was NOT. I could see that leading to a problem.

So let's start with you just checking the CF Admin, for both the cfusion instance AND the new instance, to see what you see in the "settings summary" page (on the first admin "settings" menu item) or the "system info" page (top right "i" icon in the admin). Do both report 2018,0,9? or 0,8?, or do either show some earlier number? Please do be sure to report both. (You can confirm that you are looking at each admin, either by noting the changed port, such as 8500 and 8501, or more important note that the instance name appears in the top right, just under that "i" button and the "logout" link.)

And as for the 403 errors, they can occur after updating CF (to 8 or later) and then either NOT updating the wsconfig after that, or perhaps also needing to make another set of changes related to that change in update 8. I did a blog post with more detail, at https://www.carehart.org/blog/client/index.cfm/2020/3/20/how_and_why_sites_may_break_after_Mar_2020_....

But before you may dig into that, Fabian, consider first what I ask above.

Also, one more tip: when you make your browser page test, are you doing it on the server itself, where IIS is running? If you do, then by default you should be given MORE detail from IIS about what it's unhappy about. I realize it may seem it must be about CF (since it happens only to CF pages for you), but the point is that sometimes IIS might give more detail--and the problem MAY be about an IIS configuration issue. And if you find that you do NOT get any "additional detail" while visiting a page on the server itself, that is controlled by a setting in the "error pages" feature in IIS for the site, and then see "edit feature settings" on the right (or by right-clicking in that page). The default is the 3rd option, detail for local requests, custom info for remote requests.

Bottom line, Fabian, this problem should be solvable, one way or another.

[FabianCF]

Thank you for your reply, Charlie. 
We initally installed ColdFusion 2018, set up the first website (does.work.com) and connected it to the cfusion instance via wsconfig tool. Then we made sure it's working. After that we installed the latest update available at that time. This was Update 9. After installation of Update 9 we ran the wsconfig tool again and upgraded the web server. This was our only instance/web server at that time. Then after that we ran the Lockdown Wizard and locked down cfusion instance.
All before creating the second instance.

Therefore if I look into the CF admin, both instances have the same version:
cfusion (does.work.com): 2018.0.09.318650
dnw (does-not.work.com): 2018.0.09.318650

We also ran into the 403 error issue you mentioned after updating ColdFusion 2018 to Update 9 on a different, also new server. There we fixed it according to your guide. 🙂 
However for this server when applying update 9 (while only having the cfusion instance) it worked seemlessly.
Although I know this error looks similar, I have my doubts if it's really related. Because on the other server the 403 error was printed in that typical IIS error page layout. Here it's Tomcat prompting the error.
We also checked if the secrets in server.xml and workers.properties match and we can confirm that they do for the second instance. (also for the first of course)
Just to be on the safe side we also added allowedRequestAttributesPattern=".*" to the AJP connector line in the server.xml of the second instance "dnw". However it didn't help.

For your tip: We have done tests on our clients and on the server directly. However the screenshot I attached in my inital post was from the server. So even there no IIS specific error page but rather the Tomcat error show.
Our current IIS error page setting for does-not.work.com is actually "Detailed errors" because we're not productive yet and want customers to see the error messages in their applications.
I just enabled Failed Request Tracing. I get no Error, however a warning:

I will now try what Priyank suggested.

[BKBK]

My suggestion is that you install no website on the default instance, cfusion. Use this instance, where necessary, only for the configuration or management of the other instances. This means:

1) Delete the 2 sites you have installed so far, reverting ColdFusion to its initial state. Restart ColdFusion and start anew.

2) Use the Enterprise manager to create two new instances, dnw1 and dnw2 . That is, you now have 3 instances, dnw1, dnw2 and cfusion.

3) Proceed as you did before, configuring the sites "does.work.com" and "does-not.work.com" and connecting them respectively to dnw1 and dnw2.

[Charlie Arehart]

I'll just toss in, FWIW, while that's a fine recommendation, it should not be read to imply that it's necessary. It MAY help, sure, but it also may NOT be necessary. There are indeed pluses to what BKBK is proposing. 

I just mean to add that if someone was frustrated with how things were, they may not want to hear "dump what you've done and start over". Again, it MAY help and MAY be needed, but may not be needed. And the benefits may NOT be worth that effort, though surely for some folks it would be.

[FabianCF]

Yes, I agree that this would be a too drastic step for the time being.
However I appreciate your suggestion, BKBK.

[BKBK]

Too drastic? Software design decisions are never too drastic if they are made at the beginning of the project! 🙂