An another server bites it - CF no longer starts as LocalSystem

[mikechy]

So this is the 4th or 5th time we've encountered this issue.  It always plays out the same way:

Out of the box, everything is fine.  CF starts / stops / restarts very quickly.  Over time, it takes longer and longer for CF to start.  Eventually, it won't start at all unless we change the user the service runs as from "LocalSystem" to "Administrator".  As soon as we change the user it runs as, it's like brand new.

Now, before you say "We'll there's more on your server now than when you started" or "Get more memory!", the application is the same as it was the day it launched.  No new cfcs, no new templates, nothing.  This has happened all the way back to MX 6 and no one, as far as I know, even those people who PAID Adobe to fix this, has an answer.  The current machine is running MX 8 with the updaters.

Has anyone else figured out a work around for this other than changing the user the service runs as?  We've cleared the class folders, removed old logs, everything we can think of that might delay the launching and eventually causing it to time out.

Why would it running a different user "fix" the problem?  It will eventually slow down again and you'll have to go to yet another user to keep it running.

Notice:  The times for it to become unstartable is years in all of our cases.  In this case it took about two years to get to the point it is now.

Thanks in advance for anyone with a real solution.

[ilssac]

A apologize in advance for how vague this answer will be.

I remember about three years ago, we had a server behaving like you are describing.  I don't recall how I discovered it, but I found a directory, deep in the heart of the system, that had literally tens of thousands of once use files.  It took something like five hours for windows to delete all the files that where in this directory.  I know it was something related to ColdFusion but some rare feature which we were using.  It has not happened since, so I just don't remember the specifics anymore.

[Charlie Arehart]

Mike, have you looked at and found anything informative in the \runtime\logs (note that I DO NOT mean the \logs)? There is additional info about the goings on in CF, especially around startup/shutdown, that could give you more insight into what's amiss.

Beyond that, and piggy backing on ilssac's observation, if I were in your shoes I would enable a tool like ProcessMonitor (a free tool from the sysinternals group at Microsoft), to view what I/O is happening on behalf of CF. It could be that CF is doing some reading or writing of files in a directory that's become very large, and choosing to use a new user may cause it to do that in a new directory. (I realize that you said that you cleared the classes folder, which is probably what ilssac was referring to. I'm wondering if may be something else.) Just beware that you don’t' want to leave the ProcMon tool on for very long (like more than 10-20 seconds) because it does gather up a LOT of info--you'd be amazed how much I/O happens on your machine, both in and outside of CF.

I will add that if the problem remains I would welcome the chance to help you resolve this. I love challenges like this. And while you may feel that you've paid enough people to try help you , I'll note that I have a satisfaction guarantee: if you don't feel that some portion (or even all) of the time I spend with you is valuable, then I don't want to get paid for it. More at http://www.carehart.org/consulting/. Pardon the sales pitch, to those sensitive to that. I try to give free advice often here. Sometimes, though, certain problems just can't be easily resolved over email (and when doing a shared desktop session with a phone call is much more effective.) Also, for some people, paying for help for a critical problem is not at all distasteful, so I'm just sharing an option.

/charlie

[ilssac]

carehart wrote:

(I realize that you said that you cleared the classes folder, which is probably what ilssac was referring to. I'm wondering if may be something else.)

I just wanted to clarify that, no it was not the classes folder, or any of the other folders the OP mentioned.  This was some strange folder, really, really deep in the file structure that had so many files it could not be opened in the Windows file explorer.  It was mind boggling.  We where actully surprised that the machine was even operating with a digital mill stone like this around its virtual neck.

Message was edited by: ilssac P.S.  I kind of want to say that they may have been empty file names only?  But I'm not sure enough to confidently declare that after so much time.

[Charlie Arehart]

Fair enough. Though let's be clear: I've been referring specifically to the cfclasses directory (though both Mike and I relaxed and referred to it as "classes"). To be clear, in case it will help anyone (as it can indeed fill with files), we're talking about \wwwroot\WEB-INF\cfclasses.

I can think of another, perhaps, that could fill with files perhaps unexpectedly: the \cache directory, where CFCACHE content is stored, and never removed, as far as I can recall.

But that's not really "deep". Now, we should also clarify that in the multiserver mode of deployment (multiple instances), then even these normally "high level" directories are indeed now buried deep. For instance, the cfclasses would be in:

\servers\instancename\cfusion.ear\cfusion.war\WEB-INF\cfclasses (where the .'s are -'s in the case of the "cfusion" instancename)

And the cache would be even further down, in:

\servers\instance2\cfusion.ear\cfusion.war\WEB-INF\cfusion\cache

Anyway, this is where a tool like to analyze the size or count of files in your directory structure can help. A classic in this space is Treesize, a free tool for Windows at http://www.jam-software.com/treesize_free/. There are others and for other OS's, of course.

Again, though, my proposal to use the free processmonitor tool could also help in identifying high rates of I/O happening while the server is running, or starting as in Mike's case. He may not have a "high number of files" like you do, ilssac, but instead just may have a high rate of file use.

And Mike, is it really just "starting" CF that gradually slows? Or all of CF? If the latter, I would suspect yet another possible explanation: client variables. Even if you don't "use them", if you enable them (with ClientManagement in cfapplication or in the this scope of application.cfc), then by default CF will write to the client variable repository on every page visit to any application that enables them. The default store is the registry, but even if you change that to a database, it could still be a problem, which would grow over time.

/charlie