Running CF in distributed mode with IIS inside Docker Container

[faheemitian]

We are moving CF applications to Docker containers from Windows Server 2016 servers. We are using Windows authentication so have to rely on IIS for the webserver. I am trying to configure the IIS with 64bit version of isapi_redirect.dll under mcr.microsoft.com/windows/servercore/iis:windowsservercore-ltsc2019 but getting the following error:

HTTP Error 500.0 - Internal Server Error

Calling LoadLibraryEx on ISAPI filter "C:\ColdFusion\config\wsconfig\1\isapi_redirect.dll" failed

Error details in Win Events:

TimeCreated                     Id LevelDisplayName Message

-----------                     -- ---------------- -------

8/7/2019 8:29:30 PM           2268 Error            Could not load all ISAPI filters for site 'DEFAULT WEB SITE'.  Therefore site startup aborted.

8/7/2019 8:29:30 PM           2214 Error            The HTTP Filter DLL C:\ColdFusion\config\wsconfig\1\isapi_redirect.dll failed to load.  The data is the error.

isapi_redirect.properties file has the following contents:

extension_uri= /jakarta/isapi_redirect.dll

log_file= C:\ColdFusion\config\wsconfig\1\isapi_redirect.log

log_level= info

worker_file= C:\ColdFusion\config\wsconfig\1\workers.properties

worker_mount_file= C:\ColdFusion\config\wsconfig\1\uriworkermap.properties

iis_buffer_enable= true

auth_complete= 1

iis_skip_custom_errors_enable= false

The connector is not creating any logs. The connector and configuration work fine on the Win2019 host but never inside the container. I have tried the tomcat connector and it's working fine just the CF connector is not working. Any help would be highly appreciated.

[Charlie Arehart]

The answer is in your "subject". 🙂 With IIS being in one container and CF in another, there's simply no way (out of the box) that the the named isapi_redirect.dll is going to be found on the IIS image. You have no ColdFusion2018 folder there, right?

What you are trying to achieve is what CF has indeed long called "distributed mode", as you say in your subject. With that, the web server is on one machine and CF is on another. And therefore you need to put the needed web config files on the web server, not the CF server.

But the wsconfig tool (running on the CF box) doesn't know how to do that (nor could it be expected even to be able to put files on the "other server", or in your case the "other container").

Over the years various resources have talked about how to solve this, and the same solution they show for any "other server" will apply as well to an "other container" running IIS (or Apache). Of course, one challenge is that you may want to do the steps on a host and copy the needed files into the image at build time (or container at run time).

Here are a couple of resources discussing the distributed mode process, one from Adobe with less detail, and one from Benjamin Reid with more (his is in comments, and is about IIS, despite the post being about Apache):

Setting up ColdFusion in distributed environment - ColdFusion

ColdFusion 2016 Distributed mode Apache web server - ColdFusion

I've been meaning to put something together more specifically for Docker deployment, and I will in time. Until then, hope this helps.

[faheemitian]

Charlie, thank you for getting back to me so promptly. I have actually looked at Benjamin's comment and I indeed have the wsconfig files copied over to the IIS container. Just the config folder, not full-blown CF. As I said, when I replace the isapi_redirect.dll in the IIS container with the tomcat's version, the filter loads correctly. I am wondering if isapi_redirect.dll needs anything else on the IIS server. I would really appreciate any insight into debugging the issue.

[Dave Watts]

In addition to the binary file isapi_redirect.dll, IIS needs configuration information that's really just a bunch of text files. This is typically created by wsconfig.exe when you set up the connector. The text files tell IIS how to talk to CF via the DLL - what CF instance, what port it's on, etc.

Generally, the way you set this up is to install CF on the IIS server, then just turn CF off. You could then remove most of the CF install manually if you really wanted to, but you'd obviously have to leave the text files themselves, which will be in a "wsconfig" folder if I recall correctly. I don't have an IIS-connected version of CF in front of me to verify this, but it should be easy to find. If you have multiple IIS virtual servers, and CF connected to each one individually, you'll see a bunch of subdirectories in wsconfig: wsconfig\1, wsconfig\2, etc. Otherwise you'll just have "wsconfig\1". Again, I'm working from memory here, but it should be close enough.

Dave Watts, Eidolon LLC

[Dave Watts]

Also, I really don't recommend distributed mode, because it requires a lot of configuration compared to using a reverse proxy. If you can use Apache instead of IIS, it's easier to set up a reverse proxy, but it's possible to do this with IIS if you need to. You'd need to use the built-in web server on the CF install itself.

Setup IIS with URL Rewrite as a reverse proxy for real world apps. – IIS Field Readiness – blog of t...

Dave Watts, Eidolon LLC

[faheemitian]

Thank you Dave for the advice. Unfortunately, we are using Windows Authentication and can't use Apache. Also, I have copied the entire config folder which contains the need configuration and properties fils. I might try and copy the entire CF.

[faheemitian]

Also, I have the exact same configuration with wsconfig running on a new Win2019 instance in the AWS EC2 machine. The configuration is working as expected. It just doesn't work inside the container on Windows 2019 server. The same image is even working fine on my Windows 10 just not on the Win2019 server inside the iis-server core container. It looks like there is a bug in the dll but I could be wrong. I need more information on how to debug inside the dll.

[Priyank Shrivastava. · Accepted solution]

Hi,

In the Docker Container where you are running IIS, do you have Visual C++ Redistributable pack installed?

Thanks,

Priyank

[faheemitian]

Hi Priyank,

Yes. I have Visual C++ 2012 x64 Redistributable package installed inside the container.

[Priyank Shrivastava.]

Hi,

Can you also install VC++ 2017.

Thanks,

Priyank

[faheemitian]

Same LibraryLoaEx error even after installing VC++ 2017 as well...

[Charlie Arehart]

Check to see if the application pool used by your site is running as a 32-bit pool. If so, then if you didn't tell the CF wsconfig tool to create a 32-bit connector, it created a 64-bit one.

To see the bit-level of the app pool, here's a post that discussed the same problem and solution (though for a different error, but also from a CF'ers perspective):ColdFusion vs. IIS7.5 – Application Pool

Let us know if that does it. (As for my first answer, it wasn't clear from your original question that you did already have the wsconfig files implemented within your container. That would have been one problem. This loadlibraryex is yet a different one. It didn't dawn on my to propose the bit-level suggestion because I thought yours was more about not having the needed config files in teh container at all.)

[faheemitian]

I have confirmed the bitness for the connector and the AppPool. It's all 64 bit. I also tried the configuration with 32bit version with 32bitness enabled on the AppPool still got the same error.

I could have been more descriptive in the question. My bad.

[Charlie Arehart]

So to be clear, you are confirming the bitness of the app pool in the IIS within the container, right? Not in the host, where you built and then imported into the container the files? How are you confirming it in the container, without an IIS UI? (I know that you can. I’m just trying to make sure you’re not missing something.)

/charlie

[faheemitian]

Yeap, inside the container and not the host. I have confirmed settings inside the container with both powershell and the IIS UI.

PS C:\> docker exec -it $id powershell

Windows PowerShell

Copyright (C) Microsoft Corporation. All rights reserved.

PS C:\> get-webconfiguration "/system.applicationHost/applicationPools/add[@name='DefaultAppPool']" | fl *

name                        : DefaultAppPool

queueLength                 : 1000

autoStart                   : True

enable32BitAppOnWin64       : False

managedRuntimeVersion       : v4.0

managedRuntimeLoader        : webengine4.dll

enableConfigurationOverride : True

managedPipelineMode         : Integrated

CLRConfigFile               :

passAnonymousToken          : True

startMode                   : OnDemand

state                       : Started

applicationPoolSid          : S-1-5-82-3006700770-424185619-1745488364-794895919-4004696415

processModel                : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

recycling                   : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

failure                     : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

cpu                         : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

environmentVariables        : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

workerProcesses             : Microsoft.IIs.PowerShell.Framework.ConfigurationElement

PSPath                      : MACHINE/WEBROOT/APPHOST

Location                    :

ConfigurationPathType       : Location

ItemXPath                   : /system.applicationHost/applicationPools/add[@name='DefaultAppPool']

Attributes                  : {name, queueLength, autoStart, enable32BitAppOnWin64...}

ChildElements               : {processModel, recycling, failure, cpu...}

ElementTagName              : add

Methods                     : {Start, Stop, Recycle}

Schema                      : Microsoft.IIs.PowerShell.Framework.ConfigurationElementSchema

[Dave Watts]

In that case, I'd consider using IIS to reverse proxy requests to your CF Docker instance which itself would be running the built-in web server. This would reduce dependencies like file systems needing to look the same, and wouldn't require any CF components on the IIS Docker instance.

Dave Watts, Eidolon LLC

[faheemitian]

I can give it a shot. Do you know if I can use Win Auth with URL-Rewrite and pass authenticated user info to the ColdFusion instances?

[Dave Watts]

Windows authentication via Tomcat (the built-in CF web server) is complicated. But do you really need Windows authentication there, or do you just need to know the username, or can you even get away with just knowing that the user was authenticated by IIS in the first place? If the latter, then you don't really need to worry about anything. If you just need the username, you can extract it from the Authorization variable that will be passed along by the reverse proxy as discussed here.

https://stackoverflow.com/questions/7364736/http-server-behind-iis-pass-authentication-headers

If you really need credentials on the CF server to perform additional Windows authentication requests, this might be helpful.

Apache Tomcat 8 (8.5.43) - Windows Authentication How-To

All that said, this is really getting complicated at this point. Can I ask what you're trying to accomplish by separating IIS from CF, exactly? I might be able to recommend a better solution than what I've recommended so far. (No promises there, of course!)

Dave Watts, Eidolon LLC

[faheemitian · Accepted solution]

Hi Dave,

Finally, I have it working now but let me answer your questions first. Adobe CF team has released Linux only base docker images and our applications are currently running on IIS with Windows authentication. ColdFusion is not directly accessible by the end-users. CF Apps, however, do look at the remote_user/auth_user headers to validate the user. I am thinking we can use distributed mode to use Adobe CF base images on Linux nodes and run IIS as web server for Win Auth on Win Nodes. We can create and maintain our Windows Based CF images but that will be just more work. Also, I don't want to change the application.

Reverse Proxy method you mentioned doesn't support Windows authentication since re-write actually happens before the authentication -> IIS URL Rewriting and ASP.NET Routing | Microsoft Docs

Now, for the solution, thank you Priyank for pointing me towards the VC++ redistributable. I had the command to install the redistributable in the docker file but it wasn't executing correctly and didn't error out either. I assumed it was CF error but it was it ended up to be the Docker error in the end. Also, I guess Tomcat connector doesn't rely on the redistributable and it threw me off. I had the command as:

SHELL ["powershell"]

RUN C:\temp\vcredist_x64.exe /quiet /install

It's only after I verified the installation inside the container I realized that the redistributable was not installed. *sigh*. This is not normal. I have spent over 30 hours looking into this issue. I looked into it and Microsoft recommends using CMD shell to execute the installer correctly. Powershell doesn't work well with the executable for some reason. Following syntax works:

SHELL ["powershell"]

RUN cmd.exe /s /c "c:\temp\vcredist_x64.exe /install /passive /norestart /wait"

Thank you Priyank, Dave, and Charlie. I wasn't expecting this level of support on a public forum but you guys came up with genuine recommendations. I appreciate your help.

[Charlie Arehart]

Hi, faheemitian. First, thanks for getting back to us all with the solution you'd found. That should help many, in this boat. Also, I changed your answere here to be the "correct" one (another had been selected, from before you came back with this, and it supercedes what was in the other helpful answer, from Priyank.)

Also, in case it may help others, while you had shared the isapi_redirect.properties, there's nothing really different about that versus a typical wsconfig setup. But for the sake of others working on setting up IIS in one container talking to CF in another, was there anything you may have tweaked in the workers.properties file, so that it could talk to the CF container?

And were you doing these via docker compose? might you want to share any specifics about any networking config changes you needed to use to allow the IIS container to talk to the CF one? Again, I've been meaning to better document things myself (in a series of posts), but until I may get to that, I just wonder if you may have any insights to share. Either way, again thanks for this tip.