12
Replies
Well, yeah. 
(You do need to remove any jars added during a hotfix to roll it back.) Usually the technotes for each hotfix do explain this and point out as well where the jars get put (if you use the Admin interface to upload them and therefore don't know the location yourself.) It's either \lib\updates in Standard/Server deployments, or deep in an instance in the Multiserver mode of deployment (such as C:\JRun4\servers\cfusion\cfusion-ear\cfusion-war\WEB-INF\cfusion\lib\updates).
As for applying the hotfix causing CPU and memory problems, I would doubt that a proper implementation of a hotfix could/would cause it. I suppose a mis-implementation could (putting files in the wrong place). It's quite easy to do. For instance, people often have more than one place where the CFIDE directory may live, either in their external web server docroot, or perhaps in more than one webserver docroot, or even in the docroot for CF's built-in web server.
I have asserted that the technotes ought to be updated to tell the user to look in their CF Admin, at the Mappings page, to see where IT says the CFIDE is for that instance. Because if you update the Jars for that CF server, but then mistakenly update the CFIDE in a directory other than what CF thinks you're using, I could see there being a problem.
Similarly, I could see their being a problem if you make a mistake in updating only some files, or putting them in the wrong place even within the right CFIDE.
It's definitely unfortunate that the hotfix process (with all these manual CFIDE edits) is so easily done wrong. Before anyone chimes in to decry the situation, Adobe definitely knows and there is work underway to improve the process in the future.
/charlie
Wow, so much to consider. I'll try to summarize:
- about the technotes, just to be clear, I did say that "some" say how to back it out. I wasn't sure which you were referring to and didn’t double-check myself.
- about applying both hotfixes and the jvm update at once, yep: lesson learned there, grasshopper.
- about applying either to production servers before test ones, I'll assume you learned the lesson there (and if you have no test servers, besides lamenting that, at least it would have been wiser at least to do only one at a time, right?)
- about hung threads and having no idea why, yep: cfstat shows only how many. Good that you knew that. As for what was running, you need either the CF Server monitor (if on CF Enterprise) or FusionReactor or SeeFusion if on Standard (or Enterprise).
- about "I applied the updater (rather than updating the JDK lock and stock)", I'm not sure I understand. Do you mean the CF 9.0.1 updater? That would not update the JDK version, I don't think. Certainly not to the _24 version that the oracle bug fix required.
Hope some of that helps.
BTW: if someone is ever stuck with such problems and does want assistance, there are several companies (myself included) that do that sort of thing.
/charlie arehart
charlie@carehart.org
Providing fast, remote, on-demand troubleshooting services for CF (and CFBuilder)
More at http://www.carehart.org/consulting
Oh, ok. And no offense intended, of course. I just was going on what you said:
"We did this process on a few of our web servers and then sat back to see the effect. Once both patches were installed we started having bouts of servers running out of memory within 24-48hrs and this strange phenomenon of CPU stuck at 25% or sometimes 75% or 100%. cfstat was showing hung threads but I could not identify the culprits to identify a script. Out other non patched servers were humming away just fine. I decided to rollback once I realised all our patched high-volume web servers were consistently causing problems."
So I guess you meant you did it in test and "saw no effect" and then moved to prod, where the stuff hit the fan. Of course, that begs a different question (being a "best practice" shop): are you saying you applied an equivalent load in test that you'd see in prod? If you'd say yes (and again, good for you), then it may be instead that the nature of the load you're simulating isn't really quite in line with what's really happening in prod. I see that often, with prod servers getting clobbered by spiders and bots, which are rarely accounted for in load test simulations. Or it could be that there's some unexpected difference in configuration between test and prod that could account for it.
I'm just sharing all this as much for other readers as for you, Ciaran, if you'd say that none of it is new information for you. Just trying to help.
I hear you saying " we are not the first people to have an issue with this hotfix", and fair enough. I did say in an earlier note that these new more complex hotfixes are really susceptible to mis-implementation, even by folks experienced with CF hotfixes (and most are not, so it's all the more daunting). But I'm saying I've not heard of someone having "high memory and CPU" as a result of correctly (or even incorrectly) applying these latest hotfixes. I suppose some may have. I'm not saying "no one has".
As for the server monitor being completely unresponsive, that could be a clue. Can you get in now and see if perhaps Start Memory Tracking is turned on? Or if this is 9.0.1, you can also see that in the CF Admin now (which could be useful especially if perhaps one could not get into the Server Monitor but could get to the CF admin.)
As for that updater for the JVM (itself), I'd not heard of that. Thanks for clarifying and sharing the link (http://www.oracle.com/technetwork/java/javase/fpupdater-tool-readme-305936.html), in case it may interest others.
Finally, as for the offer, it was sincere, not a sales pitch. I do understand many prefer to solve problems themselves. Still, sometimes it's more economical to get help, especially when there's no minimum commitment and a satisfaction guarantee. But sounds like you'll sort it on your own. Hope it goes well, and thanks for sharing your observations for everyone to learn from.
/charlie
Hi Charlie
I appreciate you are trying to help, it's nice to get extensive replies such as yours on the CF forums, I usually resort to StackOverflow
We find it very tough to similate real load on test machines, it's just the sheer combination of different requests, and replicating all those edge cases. We can load test certain popular request types but trying to cover every eventuality and get 100% coverage is almost impossible. To that end we try to catch unexpected problems early based on pro-active monitoring, and we did think before rolling out the first patches in production to make sure we caught problems early for each type of server we have (we run some servers as admin, others as general web and others as Flex/BlazeDS servers and yet others for mobile sites and scheduled jobs and tasks).
I'll check out the server monitor tomorrow and see about the memory settings. Thanks for that tip.
When we do get to the bottom of this I'll update here. Again thanks for the feedback.
Cheers,
Ciaran
AdChoices