Recommended heap sizes for Dev/Live/Shared CF boxes

[Owainnorth]

Hi all

I've searched through all the relevant past posts, but most of them were posted before CF8.0.1 going 64-bit, so the situation has changed slightly.

What are people's opinions on heap sizes for various server setups? I have my dev server - I leave this on 512MB and that's fine for most of the stuff I do. We generally up this to a gig on our live servers (which only run 3-4 small sites), and historically we've not gone above this on our Shared servers (which host anywhere up to 400 sites) because of the 32-bit limit.

However, now we're running CF9 on 64-bit Server 2008, we've obviously got a lot more RAM available. The obvious choice is to ramp it up high, but I've also been heard that causes issues of its own - the Java GC then has far more work to do when it's called and as the memory fills it's harder to get a contiguous block of RAM, so performance can suffer.

Does anyone have any experience of running Shared servers, or busy sites? The Server monitor is great for tuning maximum threads and the suchlike, but I can't really sit in front of it for hours waiting for the heap to fill. Also, our live servers are only Standard edition, so there's no Server Monitor.

Thoughts?
Ta

O.

[Adam Cameron.]

You could always get hold of a copy (some copies ~) of FusionReactor, which is better than the CF version anyhow.

I don't think there's a hard-and-fast rule as to what settings to use, because all apps work differently, and hit different resources differently.

I'd grab your web server logs and use those as a basis for some load testing in a dev environment, and just watch what FusionReactor says is going on, and keep an eye on any choke points.

I've not done too much of this sort of stuff, but I got the impression that one should bump up the heap size whilst it's clear that full GCs are taking place often enough to cause the app to slow down, and once the heap is big enough that the quick GCs can mostly deal with memory recycling, give it a bit more, and that's about right.  But one ought to revisit it from time to time, as traffic patterns change.

The best thing I found FusionReactor useful for was monitoring my JDBC calls: I found a lot of queries that were being re-run by accident, or just had really shit SQL in them.  Ditto with CF code that was getting knotted up unexpectedly, or re-called too many times.  Sorting out those issues helped performance more than tuning the heap size to be my perception of "ideal".

Load testing and using FusionReactor (etc) to tune one's site should be a compulsory task before any site or decent-sized update to a site goes live.  Too many people only test syntax and logic, and never load.

--

Adam

[BKBK]

I have my dev server - I leave this on 512MB and that's fine for
most of the stuff I do. We generally up this to a gig on our live
servers (which only run 3-4 small sites), and historically we've not
gone above this on our Shared servers (which host anywhere up to 400
sites) because of the 32-bit limit.

However, now we're running CF9 on 64-bit Server 2008, we've obviously got a lot more RAM available. The obvious choice is to ramp it up high, but I've also been heard that causes issues of its own - the Java GC then has far more work to do when it's called and as the memory fills it's harder to get a contiguous block of RAM, so performance can suffer.

Leave the memory setting at the default value, until testing requires you to do otherwise, or your sites unravel some unforeseen memory demand. It's a risk our SysAdmin colleagues are prepared to take. With the coming of 64-bit, they've been on the alert, and have always cattle-prodded us whenever we've been tempted to push the button to go higher.

Our defaults almost coincide with yours: 512 MB for 32-bit and just over 1GB for 64-bit. So what you yourself have just said is my own basic wisdom on the subject, too: if it aint broke, don't try to fix it.

[postscript: I work full-time at one company, part-time at another. By pure coincidence, both use the same JVM memory settings.]

[BKBK]

The Server monitor is great for tuning maximum threads and the suchlike, but I can't really sit in front of it for hours waiting for the heap to fill. Also, our live servers are only Standard edition, so there's no Server Monitor.

I have had a review of our monitoring of server memory.  My single-server experience has been with:

1) Coldfusion 8 on 512 MB and 1.2 GB JVM memory, 32-bit production servers;
2) Coldfusion 8.0.1 and Coldfusion 9 on 512 MB, 1.2 GB and 2 GB JVM memory, 64-bit test servers;
3) FusionReactor to halt requests that exceed 30 seconds (Stringent, I know, but that is dictated by the hand from above!).

These have been for sites that each max out at several thousand visitors per day. However, no two environments are the same. To you, my performance numbers will only be, at best, anecdotal evidence.

We've been using FusionReactor as spider-in-the-web, across our entire development and production platforms for years. (It had been in use before I arrived). I once tested SeeFusion, and preferred FusionReactor instead.

With the exception of a developer colleague and myself, no one seems to be aware of the existence of Coldfusion's server monitor. Apparently, the focus is on FusionReactor because it's a separate product for which people paid cash. That, I think, is an oversight.

The server monitor will tell you lots of things besides request load and memory usage. For example, it will show you bottle-necks in requests and the progress of long queries. Also, you don't have to sit and wait for all of this to happen. You may configure it to save regular snapshots, and to mail you alerts. You may then risk going away for a bit, to hibernate.

You shouldn't expect any of these tools to give you crystal-ball readings. They are only good at diagnosing what has already happened. They will give you an indication of the sites, requests and functionalities that consume the most memory, but wont tell you how to allocate memory among those resources to prevent future problems. That is something you have to work out yourself.

My advice is to do what we did. Read up as much as you can on the subject. The web will throw up plenty of articles on JVM memory allocation in Coldfusion. A good place to start is Steven Erat's blog on the performance of Coldfusion's JVM on a 64-bit server. You will soon discover that it is a complex matter. The choice of amount of memory goes hand in hand with your choice of settings for garbage collection.

[Owainnorth]

Plenty of extremely useful info in there guys, I'll be going off to do some more reading up.

I've never bothered with FusionReactor before as most of the stuff I work on are internal systems which are more about loadbalancing and reliability for uptime; performance never really factors as the machines are so over-spec'd for what they are.

Sounds like something I might be interested in for other projects though, so off to look that up too.

Thanks again people.

O.

[Adam Cameron.]

I've never bothered with FusionReactor before as most of the stuff I work on are internal systems which are more about loadbalancing and reliability for uptime; performance never really factors as the machines are so over-spec'd for what they are.

Having an overspec'ed box doesn't protect you from writing sh!t code though (and, hey, everyone writes sh!t code from time to time).  FusionReactor will help find it.

--

Adam