Server stops processing ColdFusion

[dpstucson]

One of our servers is throwing "java.lang.OutOfMemoryError: PermGen space".

Here is the original jvm.config:

java.args=-server  -Xmx4096m -Xms4096m -Dsun.io.useCanonCaches=false -XX:PermSize=128m  -XX:MaxPermSize=192m -XX:+UseParallelGC -XX:-UseGCOverheadLimit  -Dcoldfusion.rootDir={application.home}/  -Djava.security.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/coldfusion.policy   -Djava.security.auth.policy={application.home}/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/neo_jaas.policy

Made  a change to bump the -XX:MaxPermSize=192m to -XX:MaxPermSize=256m but  this morning got the error again so I bumped it again to  -XX:MaxPermSize=512m.

Am I barking up the wrong tree?

ColdFusion version 9,0,1,274733 

Update level

JRun4/servers/cfusion/cfusion-ear/cfusion-war/WEB-INF/cfusion/lib/updates/hf901-00001.jar  

server OS: Windows 2003 R2, Standar x64, SP2

webserver: IIS6

database: MSSQL 2005

[pete_freitag]

The PermGen stores compiled java class files and metadata type information about them. Take a look at the cfclasses folder on production, and see how big it is, that should give you a minimum permgen size.

Another reason the PermGen could be filling up could be if you are doing a lot of dynamic class loading. Are you doing anything funky in your code, do you use JavaLoader by chance?

[dpstucson]

The cfclasses folder on the production machine is 144MB.

As far as code I cannot tell you that as I'm just the System Admin who catches the flak when the server goes south.

[carl type3]

Hi,

I would be inclined to stop CF, delete contents of cfclasses and start CF. Having said that I do not know your application, you will know better if this procedure is detrimental or not. Why do this maybe on the files in there is broken somehow, so let them build again.

The "java.lang.OutOfMemoryError: PermGen space" is usually fairly clear as to the problem and in my mind the “knee jerk” steps you took to increase MaxPermSize would seem correct.

Since you are still have a problem tho increases have been made to the relevant error maybe some logging to see what is going on might lead to a solution. Normally I would say turn on JVM logging tho to see a PermGen issue with that can be hard. In your case some logging with the Java Dev Kit tools JCONSOL and JVISUALVM might be easier to see what is happening. Just a couple things happening for me right now so I will post more details on how to do that latter on.

Sometimes I have seen the Code Cache buffer be the issue and a “java.lang.OutOfMemoryError: PermGen” error provided in the log. Strange I know the JDK style logging will help see if that is a problem.

See your using -XX:-UseGCOverheadLimit. Personally I do not like to use that switch however guess you know why it is there and you have good reasons for running JVM that way?

Perhaps interesting to know what JVM version is in use? CF9.0.1 usually installs with 1.6.0_17, I see from CF blog 1.6.0_24 is supported.

http://blogs.adobe.com/coldfusion/2011/03/15/jdk-1-6-0_24-is-now-officially-supported/

HTH, Carl.

[carl type3]

Jconsol or Jvisualvm – part of JDK how to.

Download JDK (not JRE) from Oracle:

http://www.oracle.com/technetwork/java/javase/downloads/index.html

Install JDK

Add these to CF JVM args:

-Dcom.sun.management.jmxremote.port=N (N=port number)
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.authenticate=false

Note port number depends and there may be a firewall issue to contend with (open port etc). You know the environment so user be aware. Also note JVM args should all be in one line not carriage returned as I have shown above.

Eg JVM args:
Now I never know how to give this as the form will put spaces and carriage returns (CR) in places. I guess caveat do not just paste this entirely over your JVM args. As always backup your JVM.CONFIG file first.

##example only reader be advised
java.args=-server  -Xmx4096m -Xms4096m -

Dsun.io.useCanonCaches=false -XX:PermSize=128m  -

XX:MaxPermSize=512m -XX:+UseParallelGC -XX:-UseGCOverheadLimit

-Dcom.sun.management.jmxremote.port=8701 -

Dcom.sun.management.jmxremote.ssl=false -

Dcom.sun.management.jmxremote.authenticate=false -

Dcoldfusion.rootDir={application.home}/  -

Djava.security.policy=

{application.home}/servers/cfusion/cfusion-ear/cfusion-

war/WEB-INF/cfusion/lib/coldfusion.policy   -

Djava.security.auth.policy=

{application.home}/servers/cfusion/cfusion-ear/cfusion-

war/WEB-INF/cfusion/lib/neo_jaas.policy

When CF is started with the JMX settings in place then run jconsol and jvisualvm, connect to the port and see what is happening with PermGen, Classes, VM summary, GC stats and so on.

One other thing to be aware with JMX installed you will not be able to stop the CF instance via services.msc, stop the service and use task manager to kill the Jrun task - think you are on Enterprise so take care to kill the correct Jrun.

As is the case with any logging turn it off when you are finished. If this is entirely new to you suggest have a practice on a dev CF system first before deploying on production.

Here I have a PDF example of a system running which in part shows PermGen running within limits and close to full.

www.tassweb.com.au/carl/permgen.pdf

HTH, Carl.

[dpstucson]

Will let you know what happens with the deletion of the cfclasses.  Have  to this this when it impacts our client community the least.

[Dave Watts]

With that many compiled classes, you'll probably see better performance if you don't store them in the first place. You can disable that from within the CF Administrator by unchecking "Save class files" under Caching.

Dave Watts, CTO, Fig Leaf Software

http://www.figleaf.com/

http://training.figleaf.com/