Slow page loading after saving

Report · Nov 16, 2011

Every time I save a page, it can take up to 5 secs to load but it only occurs the first time I load the page after a save.

Upon reviewing the debugging information, I notice that the "STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN" times are significantly different. Prior to saving the file and after reloading the page the second time (and any additional time thereafter) this process is normally less than 10ms. On saving the file and the initial load, this timer is often as high as 1500ms. This occurs on any page I save HOWEVER, it only occurs on the developement server. Doing the exact same tasks on another server, the "STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN" only increases by a small fraction(instead of 10ms say 50-100ms).

What could be causing the development server to take such a significant amount of time on the "STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN" ?

Windows 2008 R2 Web Server Edition running on a ESX 4.1 host (2 Processors with 4GB of RAM assigned). CF9. Both servers are identical in regards to configuration. This just started to occur the past week. No patches or updates have been applied to either server during the past week.

The actual page times are not changed (somewhere around 50-200ms to load the page +- 50-100ms for the application.cfc)

Thanks!

Report · Nov 16, 2011

ColdFusion templates are compiled at runtime. When you change a file it needs to be recompiled. That's the extra overhead.

--

Adam

Report · Nov 16, 2011

I understand however, the difference is significate and only on one server. Working on the same file on another server, the compiling process is a fraction of a sec (compared to 5 secs).

Report · Nov 16, 2011

When you say the configs are identical, to what level do you mean? IE: same server spec & OS & CF version, or down to same JVM settings, CF settings, etc too?

Are the files on a network share instead of being stored locally?

--

Adam

Report · Nov 16, 2011

Correct. Once the original server was in place, the unit was cloned and made into a development server. So all settings with the exception of paths and server name (etc) are identical.

Report · Nov 16, 2011

OK, well I have an aswer to your question then:

What could be causing the development server to take such a significant amount of time on the "STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN"?

Magic. That can be the only cause. If the boxes are identical and yet perform differently... it's magic.

If it was the general request taking longer, then it could be the performance of connectivity to other resources (DBs, web services, etc), but that's not going to figure in those metrics you're seeing.

Have you simply tried to restart CF and see if the problem goes away?

Failing that, I'd go through the JVM, CF and JRun config and verify all the settings are still as identical as you think they are. Someone could have monkeyed with it without you knowing, or some other [unexpected event] has taken place.

Is anything going into the CF, JRun or OS log files?

--

Adam

Report · Nov 16, 2011

Ran through tab by tab in the CF administrator and there are no differences between the two servers. I have actually restarted the server several times in attempt to figure out the problem. No entries in the event logs for the OS or CF. Do not know about JRun and JVM.

One note, and maybe this could shine a light, is that we recently upgraded to CA ArcServe r16. However, I even have gone as far as removing the Agent from this machine in case some how it was related. The same agent is on other servers without a problem, so I really do not see how it could be related.

Report · Nov 17, 2011

Do not know about JRun and JVM.

Best you find out then, as they're fairly critical to this situation 😉

The jvm.config file sets how the JVM runs, and is pretty much the core file of how CF runs. For multi-server installs, it's located - by default - in the JRun4\bin\ dir. I dunno where it is on stand-alone installs, but probably something like WEB-INF/runtime or something like that (that is a wild guess). Verify they're the same on both servers, or report any differences.

The JRun logs are in JRun4/logs on a multi-server install, and... again, dunno on a stand-alone install. perhaps they're munged in with the other CF logs.

What other software is running on those machines?

BKBK mentioned the saved class files. How many files are in each server's WEB-INF/cfclasses dir?

--
Adam

Report · Nov 17, 2011

I should have mentioned early on, I am the network administrator. Not the developer. I do not know the internal merchanics of how CF works. I installed CF on the two servers and configured them. Outside reviewing the logs within the CF Web Admin tool, that is all I know.

Now to the question above:

cfclasses folder on dev has 2136 items... only 1074 on the live server. under classes, just the empty.txt file on both servers.

Under logs, I review the coldfusion-event log and other than where CF is restarted (or started at system restart) I do not see any noticeable issues. We had FusionReactor on this server months ago, and see a few calls that failed, but that has been going on for months. Something I will tackle later. Checked the coldfusion-out and coldfusionout11 logs and the same, a lot of fusion reactor crap but stuff that has been going on for months.

I went as far as doing a system wide search for files that have been changed today (looking for any date/times) that match the point of doing a save test. What I found are the following that have been modified: the actual file web page (CFM), Coldfusion-out, coldfusion-out11, ServerManager log, a random generated cache file under charting/cache and a file called software (system registry file that changes every minute or so)

aggravating...

Report · Nov 17, 2011

cfclasses folder on dev has 2136 items... only 1074 on the live server. under classes, just the empty.txt file on both servers.

That's not enough to worry about. If the dev box had tens of thousands and the prod box only a few, then there might be a difference (for the usual reason that Windows struggles with directory sizes over a certain size).

I wasn't expecting the logs to throw anything up, but good to have that confirmed.

Did you check your jvm.config files?

--
Adam

Report · Nov 17, 2011

reviewed the jvm.config file under runtime/bin on both servers and they were the same.

Report · Nov 16, 2011

Go to the Caching page in the Administrator. Uncheck the option to Save Class Files. Press the button to submit the changes. Does that make any difference?

Report · Nov 16, 2011

Unforunately... no. It is interesting that it states to leave off for development server, but it did not seem to resolve the issue. I did not restart CF (does not state to do so)

Right after a file is saved.

3579 ms

STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN

After the initial opening of the file and any time there after...

0 ms

STARTUP, PARSING, COMPILING, LOADING, & SHUTDOWN

Report · Nov 17, 2011

We should be clear about one thing, which Adam has already mentioned. What you state -- that startup, parsing,..., shutdown take more time when ColdFusion processes the page for the very first time -- is actually the expected behaviour! Your question is therefore, what could be making this time to differ significantly between 2 two 'identical' ColdFusion server setups.

I have emphasized 'identical' for a reason. It is difficult, if not practically impossible, to consturct identical setups on different machines. I can think of several situations where the ColdFusion configuration may be the same, whereas the server setups are far from identical. For example, (1) the servers are connected differently to third-party resources required by the page, such as to databases or to other files; (2) the servers have different CPU speed or power; (3) one server hosts a third-party application sharing the same JVM as ColdFusion, and so on.

Report · Nov 17, 2011

I understand that in normal scenerios it is nearly impossible for indentical setups. Maybe you missed the part that the two machines are running under ESX (virtualization) which the machine was cloned using a template. Coldfusion 9 was installed and configured with the same installer, same update and each tab was matched according (actually I believe the config file was copied from the other machine then altered for the obvious reasons). Nevertheless, I have ran through each tab once again and confirmed matching settings.

The two units run on the exact same hardware (CPU, Memory, etc). Again virtualized servers (running on a Dell R710) They both call to the same SQL Server. The actual pages are indentical (at least the pages I am using as examples). The problem is, this just started happening this past week.

I am still waiting to hear back from ArcServe as I wonder if their software is to blame. I rolled out the r16 update (was using r15) to this server and then rolled back. I do not see any services running and all of the actual application files have been removed and confirmed via explorer. I have a hard time imagining that any backup solution would cause this problem, so I still believe it is consequence that the situation occur during the same time frame. I do not want to rule it out, but at this point, I cannot see how it is related. It does not tie into IIS or CF or Java, which this issue seems to be related too.

I will check into the JVM config, but unless the setting is handled through the CF Admin, whatever is configured here is default. No other user has direct access to the server. RDS is not installed or any other remote access tools. I access the server through vCenter. The individual developers accesses the server via UNC and the Web Administration tool.

No doubt whatever the issue is, is usual... magic. but outside rebuilding the server, I need to figure out the cause and solution.

Report · Nov 17, 2011

No doubt whatever the issue is, is usual... magic. but outside rebuilding the server, I need to figure out the cause and solution.

I think a good mindset to have at this point is to stop assuming the two servers are the same, because clearly they are not. If they were the same, we'd not be having this conversation. They might have started off the same - to the best of your knowledge - but they are not the same now. So no point in dwelling on how much the same they used to be; focus on how they are now different.

I'm not having a go @ you, but it's often the case where people (and I'm no different from anyone else in this regard) get a bit snowblinded by what they "know" to be true, and set the baseline of their investigations slightly incorrectly.

As this is all very mysterious behaviour, I would also dispense with suppositions like "well that will obviously still be the same because it can't have changed". Intrinsically something is not how you expect it is, but we don't know what that is so it's illogical to be assuming anything @ this point.

--

Adam

Report · Nov 17, 2011

I agree. Obviously something is not the same. Maybe cache on the server?!?

Sorry if I was saying EVERYTHING is the same. I was trying to rule out the more obvious things like settings or hardware from the picture.

The issue seems to point to the compiling process. Is there a folder that I may need to clear out? As a development box, it would no doubt compile files more often than the live server.

Report · Nov 17, 2011

Are both servers (development and production) running the same ColdFusion license (e.g. developer, standard, enterprise, etc)?