ColdFusion Server 2021 - 500 - Internal server error.

[ColdFusion_User]

Hello Community,

I have inheritted some ColdFusion servers (2021). That has been running great so far but recently, I encountered an odd issue with one of the servers. The issue is that the server stopped processing requests on all instances, whether from websites, APIs, etc. I have tried a number of actions to get this server to start processing requests but all actions have been proven futile at this point.

Steps I've taken:

1.  rebooting the server
2. resetting the IIS (restart all coldfusion services)
3.  recreate the IIS connectors (restart all coldfusion services)
4.  Ensure all services are running
5.  Upgrading to the latest available patch (22) on all instances
6.  Tried creating a new cluster and instance
7.  I've given elevated permission to the IIS user (IUSR) for all sites
8.  Verified certs
9.  Verified Handler mappings
10.  I checked all possible logs (server, application, exception, etc) but can't find anything that would indicate the error.
11.  Check event manager logs for System and Application but don't seem to find anything that jumped oput
12.  I followed some extensive process I found using AI but still nothing.

I can't seem to open a case with adobe as I am getting the error below and the admin is no longer available.

 

What I haven't done so far is to re-intsall the IIS Roles and Features. If anyone has experienced and have a solution or can help in anyway, please reach out. Thank you

[Charlie Arehart]

I'm sorry to hear of your challenges. First you have not shown what the error is. Your screenshots show that Cf itself is working, which is a great start. I suspect it's something in your iis configuration. The error message may help us, but perhaps not. 

I'll say as well that I am confident I could have your server back up (cf site serving requests again) today, perhaps in as little as 15 minutes, via remote screenshare consulting. For more on my rates, approach, satisfaction guarantee, online calendar, email, phone, and more, see the consulting page at carehart.org. I have open slots now and into this evening. I can also help after what's offered on the calendar. 

Hope we get you back on your feet ASAP, one way or the other. 

[ColdFusion_User]

Good morning Charlie,

Thanks for reaching out. I tried uploading more screenshots but they keep getting discarded, not sure why because there's no sensitve data being captured. Let me try again.

Website Error:

 
CF services:

 
IIS Connectors

In regards to your rates, I will have to consult the business as this is an enterprise level setup and get back to you on that. I also believe it's the IIS but not sure what exactly.

 

[Charlie Arehart]

That's still not enough for us to know. It could be either an error in cf or in iis. 

First, go into iis and change the server level "error pages" feature to enable "detailed errors", so that you can see more. (You can revert it later to the default of detail for local, custom for others). 

With that detail, what's the error. We don't HAVE To have screenshot. You can just paste the text. 

Look also in the coldfusion application.log, which tracks a line for each cf error (and the exception.log, which tracks several lines per error). And BEWARE that since you're using multiple instances, each has its own log directory (or it would by default, though some mistakenly change that to have all write to one folder--a terrible idea.) And don't presume to look only in the logs folder of the instance you "see" is mapped to for the site, per that wsconfig screenshot. That's what CF thinks...but someone could have manually tweaked how IIS is configured. 

This may sound daunting or overwhelming, but it's really not...especially for someone who sees it all and helps people assess it daily like I do. 

I've tried to give you enough detail here that you MAY be able to solve things with the above alone. Some need more detail, others might find info that needs MORE explanation. We won't know what's amiss until you report back. 

I'm guessing this not production, or your folks wouldn't be making you wait to get help. Do remind them that you're honestly spending more time capturing and sharing info than we may need to fix it, not to mention the time for us to see then reply, then rinse and repeat. Have them also consider about the "hourly" rate that if we're done in 15 mins, you'll pay only a quarter of that rate. 

[ColdFusion_User]

Thanks again I am awaiting a response from them, we have redundant servers so we are in a good place for now. 

here is the detailed errors.

HTTP Error 500.19 - Internal Server Error

The requested page cannot be accessed because the related configuration data for the page is invalid.

Detailed Error Information:
Module: IIS Web Core
Notification: Unknown
Handler: Not yet determined
Error Code: 0x80070005
Config Error: Cannot read configuration file due to insufficient permissions
Config File: \\?\C:\WebSites\restapi\web.config
Requested URL: https://my_server:9443/aiops/mapping.cfc?
Physical Path:
Logon Method: Not yet determined
Logon User: Not yet determined

The app and exception logs for each instance shows no recent logs.

[Dave Watts]

OK, this looks pretty straightforward! IIS can't read that web.config. I don't know what user you're running IIS as, but it definitely needs to be able to read everything in the web root including web.config.

[Charlie Arehart]

Of course, I was writing my missive (that follows) with more details, while Dave as always cut to the chase. 🙂 If his wasn't enough of a clue for you to solve it, see what I wrote. Obviously, it took me longer to write that, and the forums here don't inform us whether someone else has offered an answer. (I sometimes check, but since I though I was responding pretty quckly I didn't bother this time!) . 🙂

[Charlie Arehart]

OK, well now that's enough detail to likely resolve things. Note what it says: IIS Cannot read the configuration file (C:\WebSites\restapi\web.config) due to insufficient permissions.  

First, you may say "there is no such file in that folder"....but IIS can't even see into it to know. As you may know, when you create a site you name its webroot, and then IIS always tries to look in that webroot for a web.config (which has site-level config details, which can be set either in the IIS UI to be saved in that file, and vice-versa).

So to whom should such be "permissions" granted? The user running the site's application pool (the pool's "identity"). You can see what app pool a site uses by right-clicking the site and choosing "manage web site>advanced properties". The app pool is the first value shown. Then go to the application pools list (top left of IIS display), which will show all app pools and a column for their identity. (Since IIS 7.5, new sites get their own app pool by default, but one can change that when creating or editing a site.)

All that said, the DEFAULT identity for a new app pool (since IIS 7) is simply "applicationpoolidentity". So we could argue THAT is the user who needs permissions to read that file. But it's both a bit more complicated and also has a simple workaround you could consider.

First, how would you change a folder's permissions to name the user? Some may know you can go into Windows File explorer to view that webroot folder (in your case, C:\WebSites\restapi), and right-click it, then choose properties, then the "security" tab, which will show WHO currently has permission.

To modify it, click "edit" then "add"...but if you just type "applicationpoolidentity" as the user, that won't work. Formally, it's a hidden user: you won't even see it if you use the "advanced" button there and its "find" option. The formal name is: iis apppool\pool name (and yes, spaces are allowed, and case does not matter--but use \ not /). So if the site was "test site" and the app pool (created by default) was "test site", and its identity showed (by default) to be "applicationpoolidentity", the actual user would be: iis apppool\test site

(BTW, when entering ANY username while adding/editing that security tab, note that there's a "locations" option...and it might default to something other than this machine itself. You need to change that to this machine itself to work with such accounts that are on the machine. Click the "locations" button there, and normally the first value shown is the local machine itself.)

You need to give it only read permissions by default. Do that, and test the site now (no restart of anything needed).

And what if you don't want to bother with identifying the identity? If you google this problem you may even see someone proposing that the user to give permissions to is "IIS_IUSRS". Technically, that's a windows "group" that contains ALL the app pool identies for ALL sites. (Again, that's defined only for the machine itself, so be sure the "location" points to that when trying to add it.) And sure, you could just use that if you aren't concerned about limiting access for only the ONE identity of the app pool for the site. (Doing this also lifts the burden of dealing with if you or anyone ever CHANGES the identity for the pool, or changes what pool the site uses--which would now have a different identity.)

Finally, there are some situations where you may find you need to need to add as well one more user (with permission to that site root folder), another special local Windows user related to IIS is: IUSR. I've never fully explored when one might be needed over/in addition to the other. But consider that as an option.

I know that's all a lot to take in. In the interest of time, I am not taking screenshots--but you can find those in other resources on the web, as this is a VERY common IIS problem with these common solutions. Again, it's a matter of seconds of effort to change these things for one who knows what they're doing and who connects the dots from that error to the solution. And I didn't even want to guess originally that this might be your problem (It could have instead been a CF error, which is why I had you check the CF logs.)

Let us know if that gets you going. If not, we can see how ot goes here or again solve it far more quickly together in a shared desktop consulting session.

[Dave Watts]

I'm not sure what you mean by "enterprise level setup". Do you have paid Adobe support? If so, try to take advantage of that by contacting them directly, if you can. That might be by phone or email. If you don't know, you might take your chances with cfsup@adobe.com (I think that's the right email address) where you'll often get informal and free support. They may be able to solve this within half an hour or less.

This problem is probably in one of the log files you mentioned above. You have a lot of instances - can you go through them and turn them all off, then turn them on one by one? (This seems unlikely, but I figured I'd ask.) My guess is that there's a conflict of some sort between the different CF services, though I can't really imagine what that would be. I'd try turning the cfusion service on last, after all the other CF services.

Finally, if you can pull the money out of petty cash or whatever, @Charlie Arehart does provide good services, and if he can fix it in fifteen minutes or so, you're only paying for that time spent rather than a full hour. If this is a production outage, I think it's worth getting him involved.

[Charlie Arehart]

Thanks for the shout-out, Dave. And you're right about the email address for Adobe support.  They don't typically respond quickly, and when I wrote originally last night it seemed urgent. Anyway, since then, we three have have made good progress toward resolution.

Oddly, this note of yours (starting, "I'm not sure what you mean" about their reference to "Enterprise setup") is showing up LATER in the list than your earlier one where you offered the quick answer to the web.config permissions issue. (For those seeing this in days to come, on the day of posting the forums here report when posts were made, in terms of "seconds ago" or "minutes ago". That's what I'm basiing my comment on. By tomorrow and beyond it will just report "days ago".) Also, sometimes the order of the replies may well change. I refreshed this page and still saw Dave's "I'm not sure" as the current last one.

Oh, fun with the forums UI! 😞

[Dave Watts]

I'm not sure why it did that either, but honestly I didn't expect to reply a second time that quickly.

[ColdFusion_User]

Thanks Dave, I was able to get someone using the email you provided. We were able to recover one of our sites but still have more work to do. I will share what was done here when the server is fully restored.

[Charlie Arehart]

Before sending that email and awaiting their reply, had you tried what I'd suggested (in my reply just before Dave's note offering that address)? Correcting the permissions as I offered might have helped quickly resolved things. 

As you'd not offered any reply to it, I can't tell if you maybe missed it. I've just been trying to help, since your first post last night.