Inspiring

Question

CF2016 Update 16 w3wp.exe crash

Forum|Forum|5 years ago
September 25, 2020
3 replies
2811 views

Folks,

I've been dealing with a problem over the past month or so with one of my developers where his application is crashing my IIS server's w3wp.exe.

Essentially, the application pool keeps crashing and restarting until it hits the limit of crashes within the timeframe allowed, and then stays down - resulting in a 503 error - website unavailable.

According to the crash dumps that I've been able to collect - the faulting module in this case is MSVCR110.dll - and it's crashing because it's trying to read from memory that it doesn't have access to.

Fault bucket 1896345861034215315, type 4
Event Name: APPCRASH
Response: Not available
Cab Id: 0

Problem signature:
P1: w3wp.exe
P2: 10.0.14393.0
P3: 57899b8a
P4: MSVCR110.dll
P5: 11.0.51106.1
P6: 5098826e
P7: c0000005
P8: 0000000000051d70
P9: 
P10: 

Faulting application name: w3wp.exe, version: 10.0.14393.0, time stamp: 0x57899b8a
Faulting module name: MSVCR110.dll, version: 11.0.51106.1, time stamp: 0x5098826e
Exception code: 0xc0000005
Fault offset: 0x0000000000051d70
Faulting process id: 0x331c
Faulting application start time: 0x01d6933ff834baae
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR110.dll
Report Id: e41b3d3a-9f47-4a72-94e0-5696029da3a3
Faulting package full name: 
Faulting package-relative application ID:

I don't think we're looking at a problem with the isapi_connector.dll being out of date, as it has the date on it that I applied the Update 16 fixes, and I definitely upgraded the connector in the wsconfig tool.

The developer, in this case is using (I believe) Wheels for his application framework (don't know if this really matters or not).

I don't think that this is related to the AJP connector changes in Tomcat, as the website loads normally when starting things up and our other instances running on the same server have not experienced this problem, but looking in the isapi_redirect log I do see a LOT of calls to jk_ajp_common.c around the times when the AppPool crashes - so, admittedly, I could be wrong there. I have gone ahead and, just in case, added allowedRequestAttributesPattern to the server.xml for that instances and set it to ".*", again - just in case.

I'm pretty stumped at this point. ANY ideas?

Thanks!

This topic has been closed for replies.

D

Dan24843164p26x

New Participant

Hello. I see this thread stopped. I am experiencing the same problem. I attempted the ideas in this thread but i haven't fixed it.

This is CF 2016. HF 16. The CF Administration panel works but CF sites that are in IIS (8.5) do not.

Charlie Arehart

Community Expert

Dan, when you say "the same problem" as mguenther1272, do you mean:

You get an error about a faulting msvcr dll?
And that it happens not in every request but only after a time?
And that you have multiple instances of cf?

If there are differences please share them. And you may want to share what error you DO get, and what steps you HAVE done, rather than say merely you've done them all. Your problem may differ in some important way.

And as I said to mguenther1272, you need not wait to wade through lots of backhand forth here over days. For a potentially complex challenge like this, it's a lot more effective to go through things in a shared desktop session together, which I offer in a consulting session. More at carehart.org/consulting. It's not free, no. But we may resolve things far more quickly than you could fear--and you won't pay for time you don't find valuable.

Or sure, we can dig in as a community here.

Btw, mguenther1272, did you ever resolve things?

/Charlie (troubleshooter, carehart. org)

D

Dan24843164p26x

New Participant

it happens on every call. and there is only one instance of cf

I have:

* (re)reinstalled the msvcr 2010 and 2012.

the application event viewer shows the following, I have to restart the application pool in IIS to try again,

Faulting application name: w3wp.exe, version: 8.5.9600.16384, time stamp: 0x5215df96
Faulting module name: MSVCR110.dll, version: 11.0.51106.1, time stamp: 0x5098826e
Exception code: 0xc0000005
Fault offset: 0x0000000000051d70
Faulting process id: 0x25b8
Faulting application start time: 0x01d87f2f2f1557ab
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: C:\Windows\SYSTEM32\MSVCR110.dll
Report Id: 77326054-eb22-11ec-8174-005056b4cee6
Faulting package full name:
Faulting package-relative application ID:

Interesting, is that this machine initially wasn't getting the CF admin panel. Then I installed HF 17. That got the admin panel worked but our code broke because of an 'elvis' operator. I cleared the INF folder and restarted and that didn't help. I took HF 17 off and i have had this problem since. And that was yesterday.

Charlie Arehart

Community Expert

That's an interesting (and from my experience, unusual) problem.

Given the log entries, it does seem right to still suspect the isapi_connector.dll. I know you said you don't think it's "out of date, as it has the date on it that I applied the Update 16 fixes", but to be clear, that's not the date it should have.

It should be a date relative to the update itself--when Adobe created it. So for update 16, it should be from July 2020. Is it? If not, do upgrade it, and check the date, then try things out.

If that's not it, it may do to remove and recreate the connector (for that site), or even to create a new site (and app pool) in iis and create a new connector for that, just in case some combination of unexpected config factors are at work. I'd hold that as a last resort.

Do keep us posted.

/Charlie (troubleshooter, carehart. org)

M

mguenther1272Author

Inspiring

My bad - I mis-typed. The isapi_connecter.dll is dated July 2020. I probably should've also said that I did tear down and rebuild both the IIS site and the ColdFusion instance - with no improvements.

Although, I don't remember if I went through and deleted everything - does deleting the CF instance from the CFADMIN page delete the corresponding wsconfig directory? I know it deleted the instance from my CF2016 directory - I just don't remember if I cleaned up the wsconfig.

I could always play with the memory settings, maybe double up the available memory for the erroring instance?

Charlie Arehart

Community Expert

Well, now that you seem to be saying you have multiple instances of cf, that is a new wrinkle.

About removing one (in the CF instance mgr), it has zero effect on the connector/wsconfig. In fact, that could be a possible source of trouble, if troubleshooting attempts/tweaks may be making the connector config convoluted.

So yes, I'd propose you remove the connector related to the site with the error. Do this in 4 steps :

Look first at the wsconfig properties (in wsconfig), to see what is the wsconfig folder number (the leftmost number on each line) which goes with the iis site ID (middle number on each line).
Then remove that site's connector in the wsconfig tool.
Then ensure that folder is gone
Then ensure also that all the iis settings for that site (pointing to cf: isapi filters, handler mappings, and Jakarta virtual directory) are also gone.

Any remnants on either side could lead to trouble.

THEN add the site back with the wsconfig tool, and test again.

If you continue to struggle, you need not go it alone. I could help remotely in perhaps as little as 15 mins. See carehart.org/consulting.

Finally, fwiw, I can't imagine any connection at all between cf heap size and such dll/w3wp.exe crashes. They each run in their own processes.

/Charlie (troubleshooter, carehart. org)

M

mguenther1272Author

Inspiring

On the possibility that the issue was with a bad Visual C++ Redistributable, I did, also, uninstall and reinstall the 2012 x86 and x64 packages for it - so that we were at the most up to date version - no dice.

BKBK

Community Expert

Hi mguenther1272

What is your Windows server version? Have you applied the latest updates to it? I ask because I think this is a Windows/IIS issue.

Hence my suggestion is along the lines of your remark about Visual C++ Redistributable.

1) Stop the ColdFusion service temporarily.

2) Apply the latest updates for your Windows server.

3) MSVCR110 represents Visual C++ 2010 SP1 Redistributable, not 2012, the version you reinstalled. So reinstall Visual C++ 2010 SP1 Redistributable, just in case your version has been corrupted.
Reinstall both the x86 and x64 versions.

In fact, while you're at it, you might as well also install or reinstall

Microsoft Visual C++ 2012 Update 4 Redistributable (x86 and x64),

Microsoft Visual C++ 2013 Redistributable (x86 and x64)

and

Microsoft Visual C++ 2015 Update 3 Redistributable (x86 and x64)

For more information and download links, see the post by "Andre for Directly" at

https://answers.microsoft.com/en-us/windows/forum/all/failed-in-w3wpexe-version-75760117514-module-name/1b1f37bb-0e7c-41a0-bba0-4748ad7712d3

4) Restart ColdFusion.

5) Configure the web server connectors.

M

mguenther1272Author

Inspiring

I update my Windows Servers every month the week after Patch Tuesday.

Ther server in question is running Windows Server 2016

I have reinstalled them, and there was no effect. I did go ahead and install the 2010 redistributable as well - that put the msvcr100.dll in to the Windows, system folders.

I don't mean to question your knowledge - but when I right click and look at the details on msvcr110.dll it says that it's part of Visual C++ 2012.

I didn't even have Visual C++ Redistributable 2010 installed on the server, and every time I search on that DLL - it comes back to the 2012 redistributable.

Deleting and recreating the wsconfig folder for the affected site has not had an effect on things either.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded