Skip to main content
J
Joop Kaashoek
Inspiring
January 16, 2012
Question

How to eradicate the 'Server is busy' errors?

  • January 16, 2012
  • 2 replies
  • 9961 views

We use Coldfusion 9.01 on Linux Redhat 64bit with Apache. Our users regularly get the folllowing error message:

Application server is busy.  Either there are too many concurrent requests or server still starting up, errorMsg: Application server is busy.  Either there are too many concurrent requests or the server still is starting up

There is a lot of info on the web about this, with advice how to solve it, but we can not seem to get rid of this problem.

We have enabled metrics logging and nothing indicates that we are running out of threads at the time we get the error. Our thread config is as follows, in the instrance's JRUN.XML file

  <service class="jrun.servlet.jrpp.JRunProxyService" name="ProxyService">

    <attribute name="activeHandlerThreads">10</attribute>

    <attribute name="backlog">500</attribute>

    <attribute name="deactivated">false</attribute>

    <attribute name="interface">*</attribute>

    <attribute name="maxHandlerThreads">390</attribute>

    <attribute name="minHandlerThreads">5</attribute>

    <attribute name="port">51003</attribute>

    <attribute name="threadWaitTimeout">300</attribute>

    <attribute name="timeout">300</attribute>

When the error happens, our metrics reporting shows the following:

01/16 11:05:12 metrics Web threads (busy/total): 0/25 Sessions: 416 Total Memory=1046592 Free=597788

So the nr of requests (presumably 25, the nr of threads, not 100% sure)  is higher than the activeHandlerThreads value (10), but not as high as the maxHandlerThreads value. The error is returned immediately, and not after the specified timeout of 5 minutes.

That is the puzzling part...

Is there any other set of configuration parameters to tweak as well?

Thanks for your help!

This topic has been closed for replies.

2 replies

A
Anonymous
January 20, 2012

I'd be careful on cranking up MaxPermSize without knowing what your heap sizes are configured for.

While this won't directly impact your problem - this should be 6-8 threads per core in your server:

<attribute name="activeHandlerThreads">10</attribute>

Also, you may want to consider turning up logging on the connector.  Should be in the cfusion/lib/wsconfig directory if it is anything like Windows.

You may also want to test out Fusion Reactor to see if it'll give any insight.

I've not messed with Apache much as a stand-alone product, but I do have some extensive work with it in WebSphere.  Quote often we'll adjust the Thread limit and Threadsperchild in httpd.conf to help with httpd performance.  I don't know if they'd be aplicable to your situation.

Also, you might want to validate that the timeout setting in httpd.conf matches that in the connector and in your CF admin settings.

J
Joop KaashoekAuthor
Inspiring
January 28, 2012

Tried lots of things with JVM tuning but that did not really solve the problem.

Now thanks to the suggestion of ke4pym we changed the MaxClients and ServerLimit in the Apache prefork module (/etc/httpd/conf/httpd.conf). The values were set at 512. The JRUN maxHandlerThreads value is 400, i.e. lower than the number of Apache clients, which could cause trouble. So now we set the MaxClients and ServerLimit to 250 and the errors are not occurring anymore.

Originally the JRun maxHandlerThreads value had been at 1000, but supposedly Linux can not create so many threads? So it was lowered to 400.

We'll do some more experimenting but if this solves the issues then it will be great.

It just makes me wonder why this is not documented anywhere.

J
Joop KaashoekAuthor
Inspiring
January 30, 2012

Never Mind, the problem is STILL occurring. So we still have not found the answer. Things tried to date

- JRun Proxy thread counts, maxHandlerThread, activeHandlerThreads

- JRun Apache connector timeout settings

- JVM memory and GC settings

- Apache threads in the prefork module

This is a mystery

C
carl type3
Legend
January 17, 2012

Suggest add other CFMetrics counters to get more details:

{listenTh},{idleTh},{delayTh},{busyTh},{totalTh},{delayRq},{droppedRq},{handledRq},{handledMs},{delayMs},{freeMemory},{totalMemory},{sessions},{sessionsInMem}

Looks like your CFMetrics is set to log these details, perhaps the issue is not with one of those counters:

{busyTh}/{totalTh}, {sessions}, {totalMemory}, {freeMemory}

While the memory looks OK from your CFMetrics results there are other java memory buffers to consider. Perhaps check in the java logs for "out of memory" type messages. What java logs? I have not used Linux / Unix for a long time so fail to recall. If I was using windows the log files to read would be in ColdFusion9\runtime\logs or \Jrun4\logs (pending Standard or Enterprise).

HTH, Carl.

J
Joop KaashoekAuthor
Inspiring
January 17, 2012

Thanks, the extra logging has been put in place, waiting for a re-occurrence of the 'server busy' problem..

C
carl type3
Legend
January 17, 2012

Something else that may help understand the CFMetrics output would be this graphical tool

http://www.adobe.com/cfusion/exchange/index.cfm?event=extensionDetail&extid=1029407

HTH, Carl.

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices