Skip to main content
Known Participant
May 27, 2008
Question

CFMX 8 suddenly hangs

  • May 27, 2008
  • 10 replies
  • 5152 views
32 Bit Win2008 Server Webedition OS, 8 GB RAM, 2x2,4 GHz Xeon IIS 7,
CFMX ColdFusion Enterprise 8.0.1(195765) Server configuration,
Java Version 1.6.0_04, Macromedia JRun 4 (Build 108487)

About once a week ColdFusion hangs completly and cf-pages are not delivered (Server Error 503).
The ColdFusion adminstrator is also not available (neither remote or on the server itself).
Static html- and aspx-pages continue to be delivered (the IIS and ASPNET Applications are available).

There are no high loads on the server on either processor or memory.
The log-files doesn't show anything to be concerned about, in fact nothing. The CF-Server hangs without issuing any log entries or windows events. In two cases, a restart of ColdFusion (without a server reboot) resulted in the same behaviour 30 minutes later.

Memory usage in CF system monitor is normaly about 180MB with peaks around 260 MB.
Normaly we got a memory usage around 200 KB in session scope by 300 sessions/users.

So all in all the server is not used very heavily and appears to be running normally.

We've got no idea what happens. The only thing we notice is a possible influence by the application pool settings in the iss7. Initially we set the application pool to the default integrated mode. The server hung after about a week.
After the second time, we changed the pool setting to classic mode hoping this might help. Again after about a week the server hung. We then went back to integrated mode and the server then hung again within 30 minutes.

Up till last week, we thought this was being caused by the ODBC drivers for the Oracle 8i DB connection. However these have been replaced by OCI drivers now which seem to be performing OK.

Unfortunately, we have only gotten the crash dump settings set up yesterday so we don't have a dump at the hung stage yet.

Has anyone any idea or experiences with CFMX8 and win2008/ISS7 that could shed light on this behaviour?

Greeting
- Thomas
    This topic has been closed for replies.

    10 replies

    Participating Frequently
    June 4, 2008
    It sounds like you've successfully tamed the beast! Congrats, and kudos to your tireless efforts and persistence.

    I'm glad you were able to work this through - let me know if you come across anything else that seems like a total and complete mystery. Sometimes just talking through it in a forum such as this can help stimulate brain cells and bring clarity to a problem.
    Participating Frequently
    June 2, 2008
    Great, thanks a lot for the update Thomas. I'm more than happy to assist with SQL query tuning or to look at code and make recommendations if it would be helpful.

    I've done a lot of similar performance tuning over the last few years for heavy OLTP databases with ColdFusion reports.

    Best of luck, and keep us posted!
    ThomasMIPAuthor
    Known Participant
    June 3, 2008
    Hello Grizzly,

    we've made some fast emergency changes to the site and are scouring the pages for possible culprits. It is not necessarily all the slow pages, there is more to it than that. Even pages with fast DB results etc. can cause the load. Others that are very similar do not. We're looking for patterns.

    At the moment at least we seem to have it under control (the worst cases are taken care of).

    Above I mentioned the SELECT @@Identity using so much memory. Sometimes you can't see the forrest for the trees! SELECT @@IDENTITY works fine as expected and returns the scalar value but SELECT @@IDENTITY FROM XXX of course returns one row for every row in the table. In the particular case above there were many, many, many rows. :-p

    Cheers
    Thomas

    September 15, 2008
    ThomasMIP - would you say you've solved your problem? What would you recommend as the best way to debug this issue for other users?

    It appears you've learned a lot from the ColdFusion Monitor tool?
    Participating Frequently
    May 30, 2008
    Very interesting! That would make sense given your symptoms.

    Your thread settings should be fine as is I would imagine. The best thing you can do to help avoid this problem would be to:

    i) Update the offending report/queries so it is more efficient. It sounds like you're wrestling with a poorly written query. Or, if it's a report that's returning many thousands of records, you may want to have your developer(s) look into re-writing the report so it is paginated in some fashion (that way only ~50 records are queried at a time, not 87,000 for example)

    ii) Update the front-end report so it prevents users from impatiently re-requesting the "heavy" report over and over and over again. For example, you could disable all form control elements on form-submit, and toss up a "Please wait..." dialogue of some kind, so they are physically unable to do anything until the report has finished.

    Generally speaking, Adobe recommends ~4 threads per CPU core. So if you have a 4-processor box, that would be 16 max concurrent threads. Strangely enough, Enterprise Edition Multi-Server installation defaults to 25 max concurrent threads. The default of 25 max threads per instance has worked well for me in the past, your mileage may vary depending on what your application is doing.
    ThomasMIPAuthor
    Known Participant
    May 31, 2008
    Hi Grizzly,

    well the weekend is shot isn't it? Don't you love our job sometimes? ;-))

    Been doing some adhoc testing. Several pages can cause the load if "provoked". All are relatively slow pages (between 2 and 4 seconds overall) and I think you're right, the developers will have to make some changes as per your suggestions.

    We'll certainly give you credit when we're through and it works. Just want to keep this thread active at the moment.

    Cheers
    Thomas
    Participating Frequently
    May 30, 2008
    Have the metrics logs given you any additional insight into thread usage? Are there any thread usage patterns leading up to these hangs?
    ThomasMIPAuthor
    Known Participant
    May 30, 2008
    Hi Grizzly,

    your tip with the metrics is worth gold. We are getting patterns on the thread usage and have this afternoon (it's late evening here now) found a partial culprit by comparing the iis logs at the peaks.

    It would seem to be "impatient users" on pages with about a 3 to four second execution (a big DB select - no transactions or insert, etc.) which is then read into a drop down. Funnily, we seem to be able to reproduce the thread peek with 1 browser / log in (meaning only one CF session) when the page is repeatedly sent without waiting for the results. One user can generate about 30 threads like that. If I extrapolate to normal usage .......

    It does seem strange though, that CF doesn't prevent one session / user on a "simple" page from using up most of the thread pool.

    What's a sensible size for max JVM Threads on a full grown server (see first post for details)? We are still at the default settings thinking Enterprise means Enterprise.

    Cheers.
    Thomas
    ThomasMIPAuthor
    Known Participant
    May 30, 2008
    BTW, forgot to say that CF does "recover" if we give it enough time. It about stalls when about 30 threads (of 50) are used, then if we leave it alone will go back to 1 to 2. Max Qeue is 1000.

    During the week it obviously doesn't because the qeue just fills up more and more with these requests in a vicious circle. It can't seem to catch up or abort with the current settings.

    Cheers
    Thomas
    Inspiring
    May 30, 2008

    I also had this same problem for about 2 months...

    In our scenario we had updated from CF7 and then encountered the
    sudden hangs..

    We finally resolved this by totally wiping the HD and starting all
    over again...For us after 2 months of trying to find the issue it was
    the easiest solution...

    From other posts I feel it was some corruption that came from the
    update but doing an uninstall of CF and a clean install never fixed
    it..So that is why we started from the beginning on the same machine.

    Good luck



    On Tue, 27 May 2008 15:30:00 +0000 (UTC), "ThomasMIP"
    <webforumsuser@macromedia.com> wrote:

    >32 Bit Win2008 Server Webedition OS, 8 GB RAM, 2x2,4 GHz Xeon IIS 7,
    > CFMX ColdFusion Enterprise 8.0.1(195765) Server configuration,
    > Java Version 1.6.0_04, Macromedia JRun 4 (Build 108487)
    >
    > About once a week ColdFusion hangs completly and cf-pages are not delivered
    >(Server Error 503).
    > The ColdFusion adminstrator is also not available (neither remote or on the
    >server itself).
    > Static html- and aspx-pages continue to be delivered (the IIS and ASPNET
    >Applications are available).
    >
    > There are no high loads on the server on either processor or memory.
    > The log-files doesn't show anything to be concerned about, in fact nothing.
    >The CF-Server hangs without issuing any log entries or windows events. In two
    >cases, a restart of ColdFusion (without a server reboot) resulted in the same
    >behaviour 30 minutes later.
    >
    > Memory usage in CF system monitor is normaly about 180MB with peaks around 260
    >MB.
    > Normaly we got a memory usage around 200 KB in session scope by 300
    >sessions/users.
    >
    > So all in all the server is not used very heavily and appears to be running
    >normally.
    >
    > We've got no idea what happens. The only thing we notice is a possible
    >influence by the application pool settings in the iss7. Initially we set the
    >application pool to the default integrated mode. The server hung after about a
    >week.
    > After the second time, we changed the pool setting to classic mode hoping this
    >might help. Again after about a week the server hung. We then went back to
    >integrated mode and the server then hung again within 30 minutes.
    >
    > Up till last week, we thought this was being caused by the ODBC drivers for
    >the Oracle 8i DB connection. However these have been replaced by OCI drivers
    >now which seem to be performing OK.
    >
    > Unfortunately, we have only gotten the crash dump settings set up yesterday so
    >we don't have a dump at the hung stage yet.
    >
    > Has anyone any idea or experiences with CFMX8 and win2008/ISS7 that could shed
    >light on this behaviour?
    >
    > Greeting
    > - Thomas
    ThomasMIPAuthor
    Known Participant
    May 30, 2008
    @ Grizzly -
    >>...ruled out the DB connections...<< -
    Well I hope so, but there could also be locking / blocking transaction problems .... ;-)
    We're running traces on the DB and hope to catch that if it occurs.

    >>...access to the application source...<< -
    We have full access to all sources on the box.

    @MattN -
    >>....updated from CF7 ...<< -
    This was a clean install on a fresh OS, etc.. There was no update involved other than the old sources that are being hosted. But yes you could be right that another clean install might just clear the situation "by itself". I hope it doesn't come to that point because then we haven't learned anything about the root cause.

    But the pressure here is increasing :-((

    Cheers
    Thomas
    ThomasMIPAuthor
    Known Participant
    May 30, 2008
    Update -

    as you can imagine we are scouring everything we can find on the server.

    One thing has shown up that we can't explain:
    Several times a day there is a very high memory usage for a query that is actually scalar (returns only one value that is an integer).
    Scenario -
    - start transaction
    - cfquery 1 - INSERT into Table A
    - cfquery 2 - SELECT @@IDENTITY FROM Table A (is SQL Server - this returns the AutoID that was just issued)
    - cfquery 3 - INSERT into Table B using Identity Value returned from cfquery 2
    - end transaction

    The second step (SELECT @@IDENTITY ....) shows 2 MEGA Bytes size in the CF Server Monitor for the query variable. This is a crazy value for one integer.

    This is not the only place we use SELECT @@ IDENTITY however the other places do not show up in the server monitor (so they don't use this crazy amount of memory for a scalar select).

    Idea are welcome.

    Have a nice weekend.
    Thomas
    Participating Frequently
    May 29, 2008
    Interesting, thanks for the update.

    It sounds like you've successfully ruled out the DB connection theory, so that's good.

    Hopefully the additional logging and diagnostic information that you're gathering will start leading us towards the actual root cause.

    Keep me posted...
    Participating Frequently
    May 29, 2008
    Thomas - the DB connection issue is pure speculation at this point. The only thing we know so far from the logs you have provided is that JRun threads are being consumed and are not released in a timely manner. This is a symptom which could have one of many causes.

    If it is related to DB connections, it is not necessarily a bug in the product, but rather, a bug in the CF application(s) you are running. I've seen ColdFusion applications in the past that are heavily reliant on there being a database connection available. For example, if client variables are being stored in the database, and every page request to the site invokes a <cfapplication> instance, than every page request will require a DB connection.

    In some cases, sites like these can have some custom error handling mechanisms put in place, where if an error if any kind is detected, the user is re-directed to a friendly error page saying something to the effect of, "Sorry, but there was a problem handling your request. Please try again." If that error page also has reference a <cfapplication> (perhaps from an Application.cfm), than sometimes, applications can find themselves in an infinite loop scenario when the database goes offline. The initial request will throw an error, and the error handler will try to redirect the user to the friendly error page, and then THAT page throws an error, so the user is once again sent to the friendly error page, and THAT page throws an error....etc etc. That can tie up threads in a hurry.

    Moral of the story is, not all applications are written in such a way that they degrade gracefully when the database server goes offline.

    It doesn't seem like that is you're problem in this particular case, but I don't think we've conclusively ruled it out yet either.

    How much do you know about the CF application(s) that you are hosting? Do you have any knowledge or insight into them, and if not, can you pull the developer(s) in to take a closer look?

    Back to the JRun metrics logs, I would strongly recommend getting that set up. If you configure the metrics to spool logs every ~10 seconds or so, you should be able to closely monitor thread usage and see exactly when the thread usage starts to climb and eventually "top out". Given that timing, you may be able to coorelate that event with some other events in the ColdFusion, JRun, or web server logs. I think it would be really helpful to know exactly what request(s) are being processed around the time this thread limit is reached. That right there could point to your "smoking gun".

    It really sounds like an infinite loop is in your application somewhere (or at the very least, some VERY long running requests). As I said, that infinite loop being related to DB connection error handling is pure speculation. There's a ton of different ways programmers can shoot themselves in the foot with infinite loops.

    Best of luck, and happy hunting!
    ThomasMIPAuthor
    Known Participant
    May 29, 2008
    Hallo Grizzly,

    just to keep you up to date -

    - We didn't find any new information in the output logs after the hang today.
    - Attempts to simulate DB Connection loss on our test / staging server showed similar behaviour but also had the following:
    05/29 15:52:59 Error [jrpp-58] - Error Executing Database Query.Connection reset by peer: socket write error The specific sequence of files included or processed is: D:\mip\mip\ad\zeiterfassung\zeiterfassung_verwaltung.cfm, line: 41
    removeOnExceptions is true for MIP_SQL. Closed the physical connection.
    05/29 15:54:44 Error [jrpp-64] - File not found: /mip/ad/person/person_entfernen.cfm The specific sequence of files included or processed is: D:\mip\mip\ad\person\person_entfernen.cfm''
    05/29 15:54:47 Error [jrpp-64] - File not found: /mip/ad/person/person_entfernen.cfm The specific sequence of files included or processed is: D:\mip\mip\ad\person\person_entfernen.cfm''
    05/29 15:57:13 Error [jrpp-57] - Error Executing Database Query.Connection reset by peer: socket write error The specific sequence of files included or processed is: D:\mip\mip\ad\zeiterfassung\zeiterfassung_uebersicht.cfm, line: 41
    removeOnExceptions is true for MIP_SQL. Closed the physical connection.

    As you can see, on the staging server the connections were forcibly closed after the exception (the behaviour we would expect). CF came back on line by itself without our having to manually do anything after the connections were reestablished (also as expected / wished).

    Since the production server does not show the removeOnExceptions entry ...? They are configured the same although the hardware is different.

    We have started the metrics every 10 seconds and installed WinDbg so that we should be able to get a dump next time.

    Cheers
    Thomas
    Participating Frequently
    May 28, 2008
    Sorry, I just noticed in your original post that you DO have Enterprise Edition. That's good.

    That being the case, you may want to take a swing at configuring JRun metrics logging. That will help us gain more visibility into thread and memory/heap usage. These sites can help you get this set up if you are new to JRun metrics logging:
    http://www.adobe.com/go/tn_19120
    http://www.bpurcell.org/blog/index.cfm?ENTRY=967&mode=entry

    And as I said, I would recommend poking around the CF Monitor to see if you can identify the trouble .CFM templates that are responsible for tying up these threads. Look for reports on long running requests, or high memory usage.

    In the past, I've seen this sort of thing crop up if for some reason the database server goes offline, and/or starts rejecting connections. Your DB server connection pool settings *may* be one place to start looking if you think it could be a DB server connectivity issue. This is pure speculation at this point however, we'll need more research and information before we can really start pointing fingers.
    ThomasMIPAuthor
    Known Participant
    May 29, 2008
    Hello Grizzly,

    >>In the past, I've seen this sort of thing crop up if for some reason the database server goes offline, and/or starts rejecting connections.<<
    Yes, and we could well imagine that this happens once in awhile.

    However immediate checks of the DBs with other applications show that the DB is available for new connections (so if it was gone then only for a "moment")
    AND
    then I would ask why CF can't / doesn't recover from this hang? Especially when the DB is there (again). That would seem to us to be a very big bug that we can't imagine being present in the product. We are thinking about moving some of out DB's into a DataCenter, I would imagine that could be much worse!

    While we are writing this we just had another hang (so it's now about every 48 hours?).

    Although we had all alerts configured to make snapshots on occurence - NO snapshot was made. That's really lousy and doesn't really serve to increase the confidence in the server monitoring does it?.

    We'll check the logs again and let you know.

    Thanks again for your time and patience up to now.

    Cheers
    Thomas
    Participating Frequently
    May 28, 2008
    Interesting. Are you running CF8 as a stand-alone server, or in multi-server mode?

    Either way, I'm interested in knowing how your JVM settings are currently configured. In a stand-alone install your JVM settings can be configured in the CF Admin. In a multi-server installation, I believe some of it lies in the CF Admin, and some of it lies in the JRun admin (port 8000).

    It would be really helpful to get more visibility into the process(es) that are responsible for tying up these threads. We can do this in one of two ways: JRun metrics logging, or the schmancy new CF Monitor that comes with CF8. I believe you *may* need Enterprise Edition for either of these features however. Are you running Enterprise Edition?
    Participating Frequently
    May 27, 2008
    Have you looked at the JRun logs, and/or the web server connector (wsconfig) logs? How about the IIS HTTPERR logs?

    Also, is there anything particularly interesting in the system event logs?

    I have a hunch that the answer is there for you somewhere, but you may not be looking at the right logs.
    ThomasMIPAuthor
    Known Participant
    May 28, 2008
    Hello Grizzly,

    thanks for replying. We've looked at the logs, see problems, but don't see the real reason why the hang is occuring.

    It seems clearly to be blocked threads with no more available to use but why are they blocked. Unfortunately we don't have the thread / memory dump that could possibly give us the information.

    We've included a bit of the logs for your "pleasure". Be very glad if you can think of anything to help.

    ----------------------------------------------------------------------
    jrun_iis6_wildcard.log
    socket shutdown failed[5756]: 10054, Connection reset by peer
    2008-05-26 16:19:29 jrISAPI[3088:2940] JRun closed connection. [5756] dropped while sending command
    2008-05-26 16:19:29 jrISAPI[3088:2940] returning error page for Connection reset by peer
    2008-05-26 16:19:43 jrISAPI[4412:2940] socket shutdown failed[5772]: 10054, Connection reset by peer
    2008-05-26 16:19:43 jrISAPI[4412:2940] JRun closed connection. [5772] dropped while sending command
    2008-05-26 16:19:43 jrISAPI[4412:2940] returning error page for Connection reset by peer
    2008-05-26 16:19:50 jrISAPI[ 888:2940] socket shutdown failed[5800]: 10054, Connection reset by peer
    2008-05-26 16:19:50 jrISAPI[ 888:2940] JRun closed connection. [5800] dropped while sending command
    2008-05-26 16:19:50 jrISAPI[ 888:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:00 jrISAPI[8048:2940] socket shutdown failed[3844]: 10054, Connection reset by peer
    2008-05-26 16:20:00 jrISAPI[8048:2940] JRun closed connection. [3844] dropped while sending command
    2008-05-26 16:20:00 jrISAPI[8048:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:02 jrISAPI[5764:2940] socket shutdown failed[3880]: 10054, Connection reset by peer
    2008-05-26 16:20:02 jrISAPI[5764:2940] JRun closed connection. [3880] dropped while sending command
    2008-05-26 16:20:02 jrISAPI[5764:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:05 jrISAPI[1212:2940] socket shutdown failed[3660]: 10054, Connection reset by peer
    2008-05-26 16:20:05 jrISAPI[1212:2940] JRun closed connection. [3660] dropped while sending command
    2008-05-26 16:20:05 jrISAPI[1212:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:07 jrISAPI[1496:2940] socket shutdown failed[3688]: 10054, Connection reset by peer
    2008-05-26 16:20:07 jrISAPI[1496:2940] JRun closed connection. [3688] dropped while sending command
    2008-05-26 16:20:07 jrISAPI[1496:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:09 jrISAPI[9632:2940] socket shutdown failed[3592]: 10054, Connection reset by peer
    2008-05-26 16:20:09 jrISAPI[9632:2940] JRun closed connection. [3592] dropped while sending command
    2008-05-26 16:20:09 jrISAPI[9632:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[ 704:2940] socket shutdown failed[3532]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[ 704:2940] JRun closed connection. [3532] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[ 704:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[8436:2940] socket shutdown failed[6256]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[1768:2940] socket shutdown failed[5460]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[8436:2940] JRun closed connection. [6256] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[5232:2940] socket shutdown failed[6488]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[1768:2940] JRun closed connection. [5460] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[8436:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[5232:2940] JRun closed connection. [6488] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[1768:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[5232:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[9372:2940] socket shutdown failed[5452]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[9104:2940] socket shutdown failed[6244]: 10054, Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[9372:2940] JRun closed connection. [5452] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[6216:2940] JRun closed connection. [6288] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[3048:2940] JRun closed connection. [5608] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[9104:2940] JRun closed connection. [6244] dropped while sending command
    2008-05-26 16:20:12 jrISAPI[9372:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:12 jrISAPI[6216:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:13 jrISAPI[3048:2940] returning error page for Connection reset by peer
    2008-05-26 16:20:13 jrISAPI[9104:2940] returning error page for Connection reset by peer
    ---------------------------------------------------------------------------------------------------------------------
    httperr.log
    2008-05-26 16:17:19 10.21.0.106 22009 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:19 10.21.0.107 54329 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:19 10.21.0.106 44323 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:34 10.197.69.87 2255 10.16.0.50 443 HTTP/1.1 GET /warp.cfm?page=http://cms.mip.intra/sixcms/detail.php?template=aktuelles - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:17:34 10.197.69.87 2256 10.16.0.50 443 HTTP/1.1 GET /warp.cfm?page=http://cms.mip.intra/sixcms/detail.php?template=aktuelles - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:17:39 10.21.0.106 54091 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:44 10.21.0.106 9924 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:54 10.21.0.106 39186 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:54 10.21.0.107 18554 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:17:59 10.21.0.106 12884 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:18:04 10.197.69.87 2257 10.16.0.50 443 HTTP/1.1 GET /mip/login.cfm - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:18:04 10.200.65.40 2698 10.16.0.50 80 HTTP/1.0 GET / - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:18:09 10.21.0.108 35998 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:18:09 10.21.0.108 50086 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:18:14 10.21.0.108 36235 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:18:19 10.21.0.106 27474 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    2008-05-26 16:18:34 10.200.65.40 2713 10.16.0.50 443 HTTP/1.0 GET /mip/ma/index.cfm - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:18:34 10.200.70.55 1369 10.16.0.50 443 HTTP/1.1 GET /warp.cfm?page=http://cms.mip.intra/sixcms/detail.php?template=aktuelles - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:19:04 10.200.70.55 1370 10.16.0.50 443 HTTP/1.1 GET /warp.cfm?page=http://cms.mip.intra/sixcms/detail.php?template=aktuelles - 3 Connection_Dropped www.mip.intra
    2008-05-26 16:19:59 10.21.0.107 44522 10.16.0.50 443 - - - - - Timer_MinBytesPerSecond -
    restart Server
    2008-05-26 16:25:53 10.21.0.106 62214 10.16.0.10 443 - - - - - Timer_ConnectionIdle -

    ------------------------------------------------------------------------------
    coldfusion-out:
    05/26 16:15:00 Information [scheduler-2] - [cfprobe___probe pumap] Executing at Mon May 26 16:15:00 CEST 2008
    java.lang.RuntimeException: Request timed out waiting for an available thread to run. You may want to consider increasing the number of active threads in the thread pool.
    at jrunx.scheduler.ThreadPool$Throttle.enter(ThreadPool.java:116)
    at jrunx.scheduler.ThreadPool$ThreadThrottle.invokeRunnable(ThreadPool.java:425)
    at jrunx.scheduler.ThreadPool$UpstreamMetrics.invokeRunnable(ThreadPool.java:266)
    at jrunx.scheduler.WorkerThread.run(WorkerThread.java:66)
    ...
    This is repeated until the server is shutdown.

    Greetings

    -Thomas