Editing CF8.0.1 Sandbox Settings Slows Down IIS7

[elarin]

Hello,
I'm running CF8.0.1 on Windows Server 2008 Standard (all 32-bit) and our web site content is located on a UNC share. When I log into ColdFusion Administrator and perform any task under Sandbox Security (i.e. adding a sandbox, editing properties of existing sandboxes), our web site comes to a crawl and becomes very unresponsive. Sometimes the behavior does not correct itself automatically forcing me to restart the WWW and CF services on the server.

Does anyone else experience such a behavior? I don't know if it's isolated to Windows 2008 or CF8.0.1. Seems like this is somewhat of a performance bug. Yes, I know I can manipulate sandboxes via adminapi. In fact, we do that most of the time. However, there are circumstances when I need to edit them within CF admin.

Overall, our performance has declined a little since we moved from ColdFusion 7.0.2 running on Windows 2003. This is somewhat disappointing.

Erick

[BKBK]

I would go to the Server Monitor section of the Coldfusion Administrator, and examine the Request Statistics and Memory Usage.

[Charlie Arehart]

Elarin, this is something I just helped solve for someone else. See his writing of the details at http://russ.michaels.me.uk/index.cfm/2009/3/19/ColdFusion-8-performance-Issues-when-using-Java-6. Bottom line, it was related to the oft-mentioned problem of a class loading bug in Java, fixed by upgrading the JVM to 1.6.10+ or back to Java 1.5. The blog entry mentioned (and many others) talk about how to do that.

My theory is that making changes in the sandbox may force Java (the sandbox security is built atop the underlying java security manager) to unload and reload the classes for CF pages in any sandboxes (I say "any", because he was seeing that changes to ANY sandbox affected performance of ALL templates, and in his case he had a sandbox defined for all CF apps.)

Let us know if that's the solution for you, Elarin.

[elarin]

Charlie,
Thank you for providing me the link to that page as it does seem I'm not the only one having this issue. I am currently running JVM 1.6.0_12 on the server. I did not keep the default 1.6.0_4 that ships with CF 8.0.1 due to the class bug.

In that page, there's mention of Cumulative Hot Fix 2 that states something about memory leaks. Perhaps I should download and install that.

Erick

[Charlie Arehart]

Hmm. Well, I will just clarify that I was with him (providing support online) while I addressed this. All he did was do the JVM update, and things were much better. He must have applied the hotfix the next day. We wondered if it was possible that the improvement we were seeing immediately may have had anything to do with the server being more responsive as much because it had just been restarted. We didn't dig in to confirm (via jvm debugging output showing class loading details) whether and how the jvm update may have helped. It could certainly be that the other fix was as or more important, so sure, please do apply it and let us know. (I would wonder if it's really the combination of the two that's key.)

/charlie

[elarin]

Well, my server took a turn for the worse a few days ago. Any web request (.cfm, .htm, .asp) would either time out or take minutes to complete. What I saw was the OS thread count for the application pool process (w3wp.exe) assigned to my site keep increasing and the jrun process sat at 0% CPU utilization. I understand CF runs under the jrun.exe process, but an application pool (w3wp.exe process) does seem to come into play for CF requests. You can see this by viewing current requests under "Worker Processes". I know this means IIS is simply getting the request first and then handing off to CF. So what I'm experiencing is a contrast to most people saying CF and IIS application pools are not related. They seem to be with IIS 7 and CF 8. I had to ditch the server and revert back to my old server with Windows 2003 and CF 7.

In conclusion, I think there's a serious compatibility issue between Windows 2008 and CF 8, maybe isolated to its 32-bit counterparts. I say this because a quick test of both 64-bit versions appears to perform as expected. However, 64-bit is not an option for us.

Erick

[BKBK]

I suspected the JVM change might not solve the problem. You're having problems with the sandbox, hence with Coldfusion's built-in objects and classes. If Coldfusion objects cause performance problems, then the obvious place to verify this is the monitor. Hence my suggestion earlier for you to go to the Server Monitor section of the Coldfusion Administrator, and examine the Request Statistics and Memory Usage.

If the monitor tells you performance is good, then the likely cause is IIS. In IIS, the likely suspect is the application pool.

[elarin]

I have good news and bad news. I found the culprit to my problem - Sandbox Security. My web server runs great when it's disabled. The moment I enable it, performance degrades quickly. The bad news is I don't know how to fix this. I need it enabled to I can restrict tags/functions at a global level. I'm thinking my main sandbox needs more files/folders added to its list. I currently have the following:

<root of my web content>
<root of my web content>\-
C:\Windows\fonts\*
D:\ColdFusion8\runtime\servers\coldfusion\SERVER-INF\temp\wwwroot-tmp\-

Just having those 4 worked with CF7. What else do I need to add for CF8?

Erick

[elarin]

I've played with the "Secured Files and Directories" section of my main sandbox and no matter what I do, I'm getting a performance hit with sandbox security enabled. It seems sandbox security with CF8 is slower than with CF7 no matter what, specifically with content residing on a UNC share. I wonder if this is a known issue and/or there's a hotfix available?!

Erick

[Charlie Arehart]

Well, Erick, are you saying you did or did not apply the Cumulative Hotfix 2 for 8.0.1, as discussed early on in this thread?

The thread has clarified from the beginning that the issue is due to Sandbox security. No new conclusion there. 🙂 But since you say it worked fine on 7 and not on 8, the 2 proposed solutions have worked for others.

Just for completeness sake, can you look in your CF Admin and confirm in the System Settings page both that you see that 1.6.0_12 jvm level (you said you're running) indicated in the "java version" field, and (if you did apply the hotfix) that you see the "update level" pointing to lib/updates/chf8010002.jar?

I'm not doubting your integrity or intelligence. 🙂 I'm just being diligent, so we don't go chasing some other problem if in fact either of these wasn't applied. If those are both applied, then we do have a curious additional cause in your case. I'd love to help see this resolved.

I have another thought I'll share separately, as this one is long enough already.

[elarin]

Ah, sorry for not including that information. Yes, I'm running JVM 1.6.0_12 and CHF2 has been applied. CHF2 does not seem to have helped. When I first posted this, I thought my performance issue was restricted to editing sandboxes in CFAdmin. But now I've confirmed the fact of having sandbox security enabled itself is harmful.

Erick

[Charlie Arehart]

Erick, assuming you get back and confirm that you have applied the jvm update and the CHF2, the next thought I have is that you could do some confirmation of what's slowing the pages down, specifically, by watching them run using a tool like FusionReactor or SeeFusion. (You said you're running on Standard, so you can't use the CF 8 Server Monitor, since it's Enterprise/Developer only.)

Both FR and SF have free trials. It may be worth your checking them out.

Besides showing what requests are running at any point in time, you can also ask each tool to show you a stack trace for a single request, which can show you exactly what line of CFML the request is running. If you repeat that and the request is stuck on the same line, now you have a smoking gun to investigate.

In working with Russ, though, we never saw it stuck on any one line, which is what led us ultimately to think outside the box and I proposed the JVM update. If that's not your issue, I wonder if you may see different information revealed in stack tracing the requests (or at least confirming for yourself the "slow" pages appearing in the request monitor tools).

They can be a great diagnostic, as sometimes you DON'T see slow requests in there running, which then tells you that the problem is somewhere else, like the web server or the web server connector, etc. (I do realize you wouldn't expect that to be the case with your problem, tied so clearly as it is to the Sandbox.)

Let us know what you think of this idea.

[Charlie Arehart]

OK, thanks for that clarification. Any thoughts on running one of the tools to tell you why requests are slow?

And I was mistaken in something I said. I had looked at your first note where you said you were running "Windows Server 2008 Standard", not CF 8 Standard. In fact, had I been thinking, the fact that you are using Sandbox Security should have clued me that you're running Enterprise, since it's in Enterprise only.

Have you tried the active requests page, to see what requests are running (if you enable "start monitoring")? And if you drill into a request while "Start Profiling" is enabled, that will show you the stack trace for that request at that moment.

For more on using the monitor, I'll point out that I did a 4-part series of articles on the CF 8 server monitor, starting at http://www.carehart.org/articles/#2007_2.

For those not on CF 8 Enterprise, again, the other tools can be valuable. Some problems are just not easily resolved without them. Again, the tools are free to try and generally very easy to install and use. I should have offered URLs for them: http://www.seefusion.com and http://www.fusion-reactor.com.

[elarin]

Thank you for your assistance so far.

I have used the Server Monitor tool in CF8 and it's not helping out. Memory usage is low and nothing is sticking out in any of the requests' stack trace. In fact, sometimes I can't view Active Requests and get kicked out of Server Monitor because CF slows down or gets backed up too much.

Using ProcessMonitor from sysinternals helped me track down the issue to the sandbox security. When sandbox security is off, I see the jrun.exe process go immediately to the required file(s) to serve out the resulting page. Remember, my content is on a UNC share. So for example, it goes to "\\myFiler\shareroot\web\application\index.cfm".

With sandbox security on, ProcessMonitor shows jrun.exe "traversing" the entire folder structure for each required file. I see it going to "\\myFiler\shareroot" then to \\myFiler\shareroot\web" and so on. So imagine a single CF template with many cfincludes to other UNC paths. And CF also traverses to files in its "ColdFusion8" program folder. I understand that's how file retrievals are implemented with sandbox security. I just feel there's an underlying I/O issue with CF8 that's maybe compounded with UNC content.

Erick

[Charlie Arehart]

OK, thanks for the clarification. Well, I really wouldn't expect an issue with memory. But if you say the stack traces don't show requests to be stuck at the same point for extended periods, ok.

Your observation about the folder traversal is interesting. Most would expect to see traversal "up" the structure when looking for the application.cfm, but this reverse order I would guess might instead reflect the fact that you either have multiple sandboxes (for the levels you see it traversing). Is that the case?

As for it going to the CF8 directory, is that the wwwroot directory, or further down into its WEB-INF.Just seems worth clarifying. That's a really good diagnostic you've done. I've not done it before myself so don't know what's normal. The UNC path issue could well be significant as well.

One last thing, though: you say "When sandbox security is off, I see the jrun.exe process go immediately to the required file(s) to serve out the resulting page." That's a little curious, in that I wouldn't expect it to "go to" each page at least to "run" the page, in that it should find the page in the template cache. But it would also look at the directory where a file lives when it did the check before each execution to see if the file source had changed. This is what the "trusted cache" option is for: if you know that the source is not changing. I would wonder if things would change dramatically for you if that was turned on, even if just temporarily for you to test things.

[elarin]

Yes, I have multiple sandboxes defined. The jrun process fetches dependent CF files located in folders like "C:\ColdFusion8\lib" and "C:\ColdFusion8\wwwroot\WEB-INF".

I'm very familiar with the Trusted Cache option as we enable it for our intranet sites. Unfortunately that's not an option with this particular server as our web developer audience is so spread out.

If you're interested Charlie, I can email you trace files from ProcessMonitor which better show the folder traversing behavior I'm seeing.

Erick

[Charlie Arehart]

But Erick, as for the trusted cache, I concluded with " I would wonder if things would change dramatically for you if that was turned on, even if just temporarily for you to test things."

Would you be willing to try it just briefly to see if it has an impact on your problem? Could help identify a solution or workaround (or bug).

BTW, as far as the fear many have over using the trusted cache, and the concern that developers need to be able to easily implement updates to code, there's a nice solution to that in CF8. Ray Camden blogged about it and how in combination with the filewatcher gateway, CF can automatically pick up changes to code and clear the cache for those specifically changed files. More here:

http://www.coldfusionjedi.com/index.cfm/2008/6/19/Clearing-individual-filesfolders-from-ColdFusion-t...

http://www.coldfusionjedi.com/index.cfm/2007/6/7/ColdFusion-8-Admin-API-and-Trusted-Cache

[elarin]

Today, I played around with turning the Trusted Cache option on with Sandbox Securtiy enabled as well. It did seem to help my performance some. Using ProcessMonitor, the reason why it's a little faster is jrun.exe process is not traversing down through folders looking for application.cfc or application.cfm templates. It still traverses down to the specified ColdFusion template and a few ColdFusion .jar files.

This option is enabled for our intranet sites and our CF admin for that area already implements a clear cache automated routine via adminapi. However, I know enabling this option on my production CF server will not fly with our web developers and managers.

So where I'm at right now is my production server is running with Sandbox Security and Trusted Cache off.

Erick

[Charlie Arehart]

Erick, I think the conclusion about why it's faster is not that "run.exe process is not traversing down through folders looking for application.cfc or application.cfm templates". Rather, with trusted cache on, it's that CF no longer looks for each template (not just application.cfc/cfm) before it executes it, if it's already in the cache.

And BTW, you could still have it looking for files, if they're not in the cache, which could happen if the cache is not sized appropriately for the number of templates that get put in there. Just being clear. In fact, I've seen people do some strange coding practices that REALLY hammered the template cache, so that even with trusted cache they were still causing that IO to the files as the template cache was clearing out files often to make room for newly requested ones. At least in CF 8 Enterprise, the Server Monitor (and Admin API) report the template cache hit ratio, so you can know if this is happening. We used to have that in CF5 and before, too, in the CFSTAT, but it reports 0 since CF6.

Anyway, just one last thing about your observation of trying to run it in production. You conclude that it just won't fly, but are you saw you are aware of an automated approach using the Admin API. Do you know if it goes the extra step of using the CF Directory Watcher event gateway? If so, there's really no reason for the devs to balk. It's an amazing clean solution. More here: http://www.adobe.com/devnet/coldfusion/articles/cacheclear.html

Hope that's helpful.

Back to your original issue, it still seems that having the sandbox on is adding a burden on your machine. We need to get to the bottom of that. It may be that there's something unique about your setup (or volume) compared to others, or perhaps there's a known issue that I'm just not aware of myself.

[elarin]

No, the clear cache adminapi routine we have on our other intranet system is scheduled to run at certain time intervals. The CF Directory Watcher event sounds impressive. However, I'm not sure how it would perform on my system which contains ~35000 CF templates.

It would be nice to know if Adobe somehow changed the underlying implementation of sandbox security from CF7 to CF8 as performance was never an issue with CF7.

Erick

[Charlie Arehart]

Sure, and like I said, it would be good to get to the bottom of the problem, but before concluding that it's necessarily about CF8, since the trusted cache helped a little, can you tell us what you see as the template cache hit ratio (per the server monitor, or Admin API). If anyone there is averse to the Server Monitor, note that you don't need to turn on any of the "start" buttons  (monitoring, profiling, or memory tracking).

Just look at the "Template Cache Status" page (under the "Request Statistics" section in the Statistics tab). I wonder if it may be sub-optimal (meaning, other than 100%). Be sure to look at it after the server's been up and running and had lots of typical traffic. Let us know what you see.

As for your observation about the effect of the directory watcher with a large number of templates, I've not heard it to be a problem. When the directory watcher came out in CF 7, many had the same concern and I seem to recall assertions that it shouldn't matter how large the directories were that were being watched. Are you willing to try it? Again, it was an answer to the concern that your developers didn't like the trusted cache option. If enabling it solves this problem, even if only some, it just seems worth at least trying.

But beyond that, let us know what you see going on in the template cache. I wonder if you may have a problem related to the loading and unloading of templates (going back to the original observation about the JVM, but looking at it more from a CF internals perspective than from a JVM perspective.)

There's got to be an answer (and something unique about your setup), as I've not heard it being a general problem (that wasn't solved by the JVM update).

As always, just trying to help.

[elarin]

Sorry for the delay in responding. The "Template Cache Hit Ratio" averaged ~99% when Trusted Cache is enabled.

So I set up the Cache Clear method for testing and it's not going to come close to satisfying our needs. I'm not exaggerating here, but it's literally going to take hours for each sweep. We have ~3500 folders in the root directory it has to search. Again, using the process monitor tool, the jrun process is literally traversing down to each file to read it. This behavior occurs when Sandbox Security is enabled.

So I'm back to Sandbox Security implementation in 8 perhaps being somewhat flawed, at least in our infrastructure. Our content resides on a NetApp FAS3050. 

Erick