Copy link to clipboard
Copied
We recently upgraded from CF8 to CF10 and we're running into some issues. We started off getting a lot of OutOfMemory errors with the default heap settings. Chaning them to 768MB/1280MB which has helped, but we're still running into occasional OOM issues. We took a heapdump as the instance was having problems today with blocked threads and noticed the heap usage was near the max so it was probably going to run out of memory soon too. Below is what the eclipse memory analyzer tool found as a possible leak suspect. Does anyone have any suggestions on where to go from here? Not sure if it's an issue with CF itself or something in the app. Thanks
One instance of "coldfusion.runtime.ClientScopeServiceImpl" loaded by "coldfusion.bootstrap.BootstrapClassLoader @ 0x120bbaf0" occupies 838,847,880 (88.33%) bytes. The memory is accumulated in one instance of "java.util.HashMap$Entry[]" loaded by "<system class loader>".
Keywords
coldfusion.runtime.ClientScopeServiceImpl
java.util.HashMap$Entry[]
coldfusion.bootstrap.BootstrapClassLoader @ 0x120bbaf0
Class Name | Shallow Heap | Retained Heap | Percentage |
---|---|---|---|
88 | 838,847,880 | 88.33% | |
48 | 838,843,112 | 88.33% | |
2,097,168 | 838,843,064 | 88.33% | |
24 | 51,409,128 | 5.41% | |
24 | 12,173,192 | 1.28% | |
24 | 10,685,760 | 1.13% | |
24 | 10,587,208 | 1.11% | |
24 | 10,443,664 | 1.10% | |
24 | 9,886,280 | 1.04% | |
24 | 9,723,216 | 1.02% | |
24 | 9,254,288 | 0.97% | |
24 | 6,494,488 | 0.68% | |
24 | 6,178,320 | 0.65% | |
24 | 4,008,048 | 0.42% | |
24 | 3,542,552 | 0.37% | |
24 | 3,503,072 | 0.37% | |
24 | 2,908,120 | 0.31% | |
24 | 2,197,056 | 0.23% | |
24 | 2,050,528 | 0.22% | |
24 | 2,034,224 | 0.21% | |
24 | 1,846,808 | 0.19% | |
24 | 1,846,784 | 0.19% | |
24 | 1,846,784 | 0.19% | |
| 480 | 162,619,520 | 0.171 |
Label | Number of Objects | Used Heap Size | Retained Heap Size |
---|---|---|---|
java.util.HashMap$Entry First 10 of 190,465 objects | 190,465 | 4,571,160 | 836,711,024 |
java.lang.String First 10 of 253 objects | 253 | 6,072 | 26,408 |
java.lang.reflect.Method First 10 of 57 objects | 57 | 4,560 | 4,560 |
coldfusion.runtime.ClientScope All 1 objects | 1 | 56 | 2,632 |
java.lang.Class[] First 10 of 73 objects | 73 | 1,248 | 1,248 |
coldfusion.runtime.ClientScopeKey All 1 objects | 1 | 24 | 24 |
java.lang.Class All 2 objects | 2 | 0 | 0 |
| 190,852 | 4,583,120 | 836,745,896 |
Copy link to clipboard
Copied
Hi,
Please share the following details:
1. Settings summary
2. jvm.config
3. How frequently you are facing this issue?
4. How much time it take to throw error after restarting the ColdFusion server?
Regards,
Nimit
Copy link to clipboard
Copied
Hi Sharma,
Here's the settings summary from the console. The OOM issue is now happening about 1-2 times weekly after increasing the heap size. Before that it was almost daily. It seems we can go about 75-85 hours before we restart due to either OOM or blocked threads.
Settings Summary
Server Settings > Settings Summary
Report generated on Sep 25, 2014 12:08 PM
This report shows the status of all ColdFusion configuration settings. To display the area of the ColdFusion Administrator where you can edit the group settings, click any of the groups in the report.
System Information
Server Details
Server Product ColdFusion
Version ColdFusion 10,286680
Edition Enterprise
Operating System Windows 2003
OS Version 5.2
Update Level /D:/ColdFusion10/cfusion/lib/updates/chf10000012.jar
Adobe Driver Version 4.1 (Build 0001)
JVM Details
Java Version 1.7.0_15
Java Vendor Oracle Corporation
Java Vendor URL http://java.oracle.com/
Java Home D:\ColdFusion10\jre
Java File Encoding Cp1252
Java Default Locale en_US
File Separator \
Path Separator ;
Line Separator Chr(13)
User Name SYSTEM
User Home C:\Documents and Settings\Default User
User Dir D:\ColdFusion10\cfusion\bin
Java VM Specification Version 1.7
Java VM Specification Vendor Oracle Corporation
Java VM Specification Name Java Virtual Machine Specification
Java VM Version 23.7-b01
Java VM Vendor Oracle Corporation
Java VM Name Java HotSpot(TM) Server VM
Java Specification Version 1.7
Java Specification Vendor Oracle Corporation
Java Specification Name Java Platform API Specification
Java Class Version 51.0
CF Server Java Class Path ;D:/ColdFusion10/cfusion/lib/updates/chf10000012.jar; D:/ColdFusion10/cfusion/lib/ant-launcher.jar; D:/ColdFusion10/cfusion/lib/ant.jar; D:/ColdFusion10/cfusion/lib/antlr-2.7.6.jar; D:/ColdFusion10/cfusion/lib/apache-solr-core.jar; D:/ColdFusion10/cfusion/lib/apache-solr-solrj.jar; D:/ColdFusion10/cfusion/lib/asm-all-3.1.jar; D:/ColdFusion10/cfusion/lib/asn1.jar; D:/ColdFusion10/cfusion/lib/axis.jar; D:/ColdFusion10/cfusion/lib/backport-util-concurrent.jar; D:/ColdFusion10/cfusion/lib/bcel-5.1-jnbridge.jar; D:/ColdFusion10/cfusion/lib/bcel.jar; D:/ColdFusion10/cfusion/lib/bcmail-jdk14-139.jar; D:/ColdFusion10/cfusion/lib/bcprov-jdk14-139.jar; D:/ColdFusion10/cfusion/lib/cdo.jar; D:/ColdFusion10/cfusion/lib/cdohost.jar; D:/ColdFusion10/cfusion/lib/certj.jar; D:/ColdFusion10/cfusion/lib/cf-acrobat.jar; D:/ColdFusion10/cfusion/lib/cf-assembler.jar; D:/ColdFusion10/cfusion/lib/cf-logging.jar; D:/ColdFusion10/cfusion/lib/cf4was.jar; D:/ColdFusion10/cfusion/lib/cf4was_ae.jar; D:/ColdFusion10/cfusion/lib/cfusion-req.jar; D:/ColdFusion10/cfusion/lib/cfusion.jar; D:/ColdFusion10/cfusion/lib/chart.jar; D:/ColdFusion10/cfusion/lib/clibwrapper_jiio.jar; D:/ColdFusion10/cfusion/lib/commons-beanutils-1.8.0.jar; D:/ColdFusion10/cfusion/lib/commons-codec-1.3.jar; D:/ColdFusion10/cfusion/lib/commons-collections-3.2.1.jar; D:/ColdFusion10/cfusion/lib/commons-compress-1.0.jar; D:/ColdFusion10/cfusion/lib/commons-digester-2.0.jar; D:/ColdFusion10/cfusion/lib/commons-discovery-0.4.jar; D:/ColdFusion10/cfusion/lib/commons-httpclient-3.1.jar; D:/ColdFusion10/cfusion/lib/commons-lang-2.4.jar; D:/ColdFusion10/cfusion/lib/commons-logging-1.1.1.jar; D:/ColdFusion10/cfusion/lib/commons-logging-api-1.1.1.jar; D:/ColdFusion10/cfusion/lib/commons-net-3.0.1.jar; D:/ColdFusion10/cfusion/lib/commons-vfs2-2.0.jar; D:/ColdFusion10/cfusion/lib/crystal.jar; D:/ColdFusion10/cfusion/lib/derby.jar; D:/ColdFusion10/cfusion/lib/derbyclient.jar; D:/ColdFusion10/cfusion/lib/derbynet.jar; D:/ColdFusion10/cfusion/lib/derbyrun.jar; D:/ColdFusion10/cfusion/lib/derbytools.jar; D:/ColdFusion10/cfusion/lib/dom4j-1.6.1.jar; D:/ColdFusion10/cfusion/lib/dpHibernate.jar; D:/ColdFusion10/cfusion/lib/ehcache-core-2.5.1.jar; D:/ColdFusion10/cfusion/lib/ehcache-web-2.0.4.jar; D:/ColdFusion10/cfusion/lib/esapi-2.0.1.jar; D:/ColdFusion10/cfusion/lib/EWSAPI-1.1.5.jar; D:/ColdFusion10/cfusion/lib/FCSj.jar; D:/ColdFusion10/cfusion/lib/flashgateway.jar; D:/ColdFusion10/cfusion/lib/flex-messaging-common.jar; D:/ColdFusion10/cfusion/lib/flex-messaging-core.jar; D:/ColdFusion10/cfusion/lib/flex-messaging-opt.jar; D:/ColdFusion10/cfusion/lib/flex-messaging-proxy.jar; D:/ColdFusion10/cfusion/lib/flex-messaging-remoting.jar; D:/ColdFusion10/cfusion/lib/flex-rds-server.jar; D:/ColdFusion10/cfusion/lib/geronimo-stax-api_1.0_spec-1.0.1.jar; D:/ColdFusion10/cfusion/lib/hibernate3.jar; D:/ColdFusion10/cfusion/lib/httpclient-4.1.1.jar; D:/ColdFusion10/cfusion/lib/httpclient-cache-4.1.1.jar; D:/ColdFusion10/cfusion/lib/httpclient.jar; D:/ColdFusion10/cfusion/lib/httpcore_4.1.2.jar; D:/ColdFusion10/cfusion/lib/httpmime-4.1.1.jar; D:/ColdFusion10/cfusion/lib/ib6addonpatch.jar; D:/ColdFusion10/cfusion/lib/ib6core.jar; D:/ColdFusion10/cfusion/lib/ib6http.jar; D:/ColdFusion10/cfusion/lib/ib6swing.jar; D:/ColdFusion10/cfusion/lib/ib6util.jar; D:/ColdFusion10/cfusion/lib/im.jar; D:/ColdFusion10/cfusion/lib/iText.jar; D:/ColdFusion10/cfusion/lib/iTextAsian.jar; D:/ColdFusion10/cfusion/lib/izmado.jar; D:/ColdFusion10/cfusion/lib/jai_codec.jar; D:/ColdFusion10/cfusion/lib/jai_core.jar; D:/ColdFusion10/cfusion/lib/jai_imageio.jar; D:/ColdFusion10/cfusion/lib/jakarta-oro-2.0.6.jar; D:/ColdFusion10/cfusion/lib/jakarta-slide-webdavlib-2.1.jar; D:/ColdFusion10/cfusion/lib/java-xmlbuilder-0.4.jar; D:/ColdFusion10/cfusion/lib/javasysmon-0.3.3.jar; D:/ColdFusion10/cfusion/lib/jax-qname.jar; D:/ColdFusion10/cfusion/lib/jaxb-api.jar; D:/ColdFusion10/cfusion/lib/jaxb-impl.jar; D:/ColdFusion10/cfusion/lib/jaxb-libs.jar; D:/ColdFusion10/cfusion/lib/jaxb-xjc.jar; D:/ColdFusion10/cfusion/lib/jaxrpc.jar; D:/ColdFusion10/cfusion/lib/jcifs-1.3.15.jar; D:/ColdFusion10/cfusion/lib/jdom.jar; D:/ColdFusion10/cfusion/lib/jeb.jar; D:/ColdFusion10/cfusion/lib/jersey-core.jar; D:/ColdFusion10/cfusion/lib/jersey-server.jar; D:/ColdFusion10/cfusion/lib/jersey-servlet.jar; D:/ColdFusion10/cfusion/lib/jets3t-0.8.1.jar; D:/ColdFusion10/cfusion/lib/jetty-continuation-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-http-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-io-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-security-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-server-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-servlet-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-servlets-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-util-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jetty-xml-7.0.0.v20091005.jar; D:/ColdFusion10/cfusion/lib/jintegra.jar; D:/ColdFusion10/cfusion/lib/jnbcore.jar; D:/ColdFusion10/cfusion/lib/jpedal.jar; D:/ColdFusion10/cfusion/lib/js.jar; D:/ColdFusion10/cfusion/lib/jsch-0.1.44m.jar; D:/ColdFusion10/cfusion/lib/jsr107cache.jar; D:/ColdFusion10/cfusion/lib/jsr311-api-1.1.1.jar; D:/ColdFusion10/cfusion/lib/jta.jar; D:/ColdFusion10/cfusion/lib/jutf7-0.9.0.jar; D:/ColdFusion10/cfusion/lib/ldap.jar; D:/ColdFusion10/cfusion/lib/ldapbp.jar; D:/ColdFusion10/cfusion/lib/log4j-1.2.15.jar; D:/ColdFusion10/cfusion/lib/lucene-analyzers-3.4.0.jar; D:/ColdFusion10/cfusion/lib/lucene-core-3.4.0.jar; D:/ColdFusion10/cfusion/lib/lucene-highlighter-3.4.0.jar; D:/ColdFusion10/cfusion/lib/lucene-memory-3.4.0.jar; D:/ColdFusion10/cfusion/lib/lucenedemo.jar; D:/ColdFusion10/cfusion/lib/macromedia_drivers.jar; D:/ColdFusion10/cfusion/lib/mail.jar; D:/ColdFusion10/cfusion/lib/metadata-extractor-2.4.0-beta-1.jar; D:/ColdFusion10/cfusion/lib/mlibwrapper_jai.jar; D:/ColdFusion10/cfusion/lib/msapps.jar; D:/ColdFusion10/cfusion/lib/mysql-connector-java-commercial-5.1.17-bin.jar; D:/ColdFusion10/cfusion/lib/namespace.jar; D:/ColdFusion10/cfusion/lib/nekohtml.jar; D:/ColdFusion10/cfusion/lib/netty-3.2.5.Final.jar; D:/ColdFusion10/cfusion/lib/ooxml-schemas.jar; D:/ColdFusion10/cfusion/lib/pdfencryption.jar; D:/ColdFusion10/cfusion/lib/poi-contrib.jar; D:/ColdFusion10/cfusion/lib/poi-ooxml-schemas.jar; D:/ColdFusion10/cfusion/lib/poi-ooxml.jar; D:/ColdFusion10/cfusion/lib/poi-scratchpad.jar; D:/ColdFusion10/cfusion/lib/poi.jar; D:/ColdFusion10/cfusion/lib/portlet_20.jar; D:/ColdFusion10/cfusion/lib/postgresql-8.3-604.jdbc3.jar; D:/ColdFusion10/cfusion/lib/quartz.jar; D:/ColdFusion10/cfusion/lib/relaxngDatatype.jar; D:/ColdFusion10/cfusion/lib/ri_generic.jar; D:/ColdFusion10/cfusion/lib/rome-cf.jar; D:/ColdFusion10/cfusion/lib/saaj.jar; D:/ColdFusion10/cfusion/lib/saxon9he.jar; D:/ColdFusion10/cfusion/lib/serializer.jar; D:/ColdFusion10/cfusion/lib/slf4j-api-1.5.6.jar; D:/ColdFusion10/cfusion/lib/slf4j-log4j12-1.5.6.jar; D:/ColdFusion10/cfusion/lib/smack.jar; D:/ColdFusion10/cfusion/lib/smpp.jar; D:/ColdFusion10/cfusion/lib/STComm.jar; D:/ColdFusion10/cfusion/lib/tagsoup-1.2.jar; D:/ColdFusion10/cfusion/lib/tika-core-0.6.jar; D:/ColdFusion10/cfusion/lib/tika-parsers-0.6.jar; D:/ColdFusion10/cfusion/lib/tools.jar; D:/ColdFusion10/cfusion/lib/tt-bytecode.jar; D:/ColdFusion10/cfusion/lib/wc50.jar; D:/ColdFusion10/cfusion/lib/webchartsJava2D.jar; D:/ColdFusion10/cfusion/lib/wsdl4j-1.6.2.jar; D:/ColdFusion10/cfusion/lib/wsrp4j-commons-0.5-SNAPSHOT.jar; D:/ColdFusion10/cfusion/lib/wsrp4j-producer.jar; D:/ColdFusion10/cfusion/lib/xalan.jar; D:/ColdFusion10/cfusion/lib/xercesImpl.jar; D:/ColdFusion10/cfusion/lib/xml-apis.jar; D:/ColdFusion10/cfusion/lib/xmlbeans-2.3.0.jar; D:/ColdFusion10/cfusion/lib/xmpcore.jar; D:/ColdFusion10/cfusion/lib/xsdlib.jar; D:/ColdFusion10/cfusion/lib/; D:/ColdFusion10/cfusion/lib/axis2/axiom-api-1.2.13.jar; D:/ColdFusion10/cfusion/lib/axis2/axiom-dom-1.2.13.jar; D:/ColdFusion10/cfusion/lib/axis2/axiom-impl-1.2.13.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-adb-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-adb-codegen-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-codegen-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-jaxws-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-kernel-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-transport-http-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/axis2-transport-local-1.7.0.jar; D:/ColdFusion10/cfusion/lib/axis2/commons-fileupload-1.2.jar; D:/ColdFusion10/cfusion/lib/axis2/commons-io-1.4.jar; D:/ColdFusion10/cfusion/lib/axis2/geronimo-ws-metadata_2.0_spec-1.1.2.jar; D:/ColdFusion10/cfusion/lib/axis2/httpcore-4.0.jar; D:/ColdFusion10/cfusion/lib/axis2/neethi-3.0.2.jar; D:/ColdFusion10/cfusion/lib/axis2/woden-api-1.0.jar; D:/ColdFusion10/cfusion/lib/axis2/woden-impl-commons-1.0.jar; D:/ColdFusion10/cfusion/lib/axis2/woden-impl-dom-1.0.jar; D:/ColdFusion10/cfusion/lib/axis2/wsdl4j-1.6.2.jar; D:/ColdFusion10/cfusion/lib/axis2/wstx-asl-3.2.9.jar; D:/ColdFusion10/cfusion/lib/axis2/XmlSchema-1.4.8.jar; D:/ColdFusion10/cfusion/lib/axis2/; D:/ColdFusion10/cfusion/gateway/lib/examples.jar; D:/ColdFusion10/cfusion/gateway/lib/; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/cfgatewayadapter.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/concurrent.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/flex/jars/; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-awt-util.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-css.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-ext.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-transcoder.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/batik-util.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/commons-discovery.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/commons-logging.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/concurrent.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/flex.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jakarta-oro-2.0.7.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jcert.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jnet.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/jsse.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/oscache.jar; D:/ColdFusion10/cfusion/wwwroot/WEB-INF/cfform/jars/;
Java Class Path D:\\ColdFusion10\\cfusion\lib\oosdk\lib;
D:\\ColdFusion10\\cfusion\lib\oosdk\classes;
D:\ColdFusion10\cfusion\bin\..\runtime\bin\tomcat-juli.jar;
D:\ColdFusion10\cfusion\bin\cf-bootstrap.jar
Java Ext Dirs D:\ColdFusion10\jre\lib\ext;C:\WINDOWS\Sun\Java\lib\ext
Printer Details
Default Printer
Printers
Server Information
General Settings
Timeout requests Yes
Enable Per App Settings Yes
Request Time Limit 120 seconds
Use UUID for CFToken Yes
Disable Service Factory No
Protect serialized JSON No
Protect Serialized JSON Prefix //
Missing Template Handler /404b.cfm
Site-wide Error Handler /errorCatch.cfm
Enable HTTP status codes Yes
Enable Global Script Protection Yes
ORMSearch Index Directory
Default CFForm ScriptSrc Directory /CFIDE/scripts/
Google Map Key
Maximum size of post data 100 MB
Request Throttle Threshold 4 MB
Request Throttle Memory 200 MB
Request Tuning
Simultaneous request limit 20
Flash Remoting request limit 5
Web Service request limit 20
CFC request limit 20
CFThread Pool Size 10
Maximum number of report threads 8
Request Queue Timeout 120 seconds
Request Queue Timeout Page
Caching
Template cache size 150 templates
Enable trusted cache No
Cached query limit 500
Save Class Files Yes
Client Variable Settings
Default client variable store ecom
Purge Interval 1 hours 7 minutes
Client Stores
Registry
Type REGISTRY
Description System registry.
Purge data after time limit Yes
Time limit 90 days
Disable global updates No
ecom
Type JDBC
Description
Purge data after time limit No
Time limit 2 days
Disable global updates Yes
Cookie
Type COOKIE
Description Client based text file.
Purge data after time limit Yes
Time limit 10 days
Disable global updates No
Memory Variables
J2EE Sessions No
Application Variables
Enable Application Variables Yes
Default Timeout 2,0,0,0
Maximum Timeout 2,0,0,0
Session Variables
Enable session variables Yes
Default Timeout 0,0,20,0
Maximum Timeout 2,0,0,0
ColdFusion Mappings
/gateway D:\ColdFusion10\cfusion\gateway\cfc
/CFIDE D:\ColdFusion10\cfusion\wwwroot\CFIDE
Mail Connection Settings
Default Server Port 25
Connection Timeout 60 seconds
Spool Interval 30 seconds
Mail Delivery Threads 10
Maintain Connection to Mail Server Yes
Spool Messages To disk
Max Messages Spooled to Memory 50000
Default CFMail Charset UTF-8
Use SSL Connection No
Use TLS No
Default Mail Server
smtp.example.com
Server smtp.example.com
Port 25
Username
Password
Backup Mail Servers
smtp-backup.example.com
Server smtp-backup.example.com
Port 25
Username
Password
Mail Logging Settings
Log Severity warning
Log all E-mail messages sent by ColdFusion Yes
Charting
Cache Type disk images
Maximum number of images in cache 50 images
Maximum number of charting threads 4
Disk cache location D:\ColdFusion10\cfusion\charting\cache
Data & Services
Database Data Sources
GES
CF data source name GES
Description
Driver MSSQLServer
JDBC URL jdbc:macromedia:sqlserver://MSSQL:1433;databaseName=GES;SelectMethod=direct;sendStringParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100
Username sa
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
cfcodeexplorer
CF data source name cfcodeexplorer
Description
Driver Apache Derby Embedded
JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\cfcodeexplorer;create=false
Username
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
ecom
CF data source name ecom
Description
Driver
JDBC URL jdbc:macromedia:oracle://10.10.120.13:1521;serviceName=ecom;alternateServers=(10.10.120.1 4:1521);loadBalancing=true
Username user
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
cfbookclub
CF data source name cfbookclub
Description
Driver Apache Derby Embedded
JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\bookclub;create=false
Username
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
ecomd
CF data source name ecomd
Description
Driver Oracle
JDBC URL jdbc:macromedia:oracle://ORACLE:1521;SID=ecomd;AuthenticationMethod=userIDPassword;sendStringParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100
Username ges
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
GES_New
CF data source name GES
Description
Driver MSSQLServer
JDBC URL jdbc:macromedia:sqlserver://MSSQL\MSSQL:1433;databaseName=GES;SelectMethod=direct;sendStringParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100
Username user
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
E1_Oracle
CF data source name E1
Description
Driver Oracle
JDBC URL jdbc:macromedia:oracle://ORACLE:1516;SID=E1;AuthenticationMethod=userIDPassword;sendStringParametersAsUnicode=false;querytimeout=0;MaxPooledStatements=100
Username user
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
cfdocexamples
CF data source name cfdocexamples
Description
Driver Apache Derby Embedded
JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\cfdocexamples;create=false
Username
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
cfartgallery
CF data source name cfartgallery
Description
Driver Apache Derby Embedded
JDBC URL jdbc:derby:D:\ColdFusion10\cfusion\db\artgallery;create=false
Username
Login timeout 30 seconds
Long text buffer size 64000
Timeout 1200 seconds
Maintain connections Yes
Interval 420 seconds
Restricted SQL operations
Disable connections No
Web Services
Debugging & Logging
Debugging Settings
Enable debugging No
Enable Robust Exception Information No
Display format classic.cfm
Execution times Yes
Execution time format summary
Execution time highlight threshold 250 ms
Database activity Yes
Exception information Yes
Tracing information Yes
Timer Information No
Variables Yes
Variables
Application Yes
CGI Yes
Client Yes
Cookie Yes
Form Yes
Request Yes
Server No
Session No
URL Yes
Debugging IP Addresses
Debugging IP Address Restrictions
Line Debugger Settings
Allow Line Debugging NO
Debugger Port 5005
Max Simultaneous Debugging Sessions 5
Logging Settings
Log directory D:\ColdFusion10\cfusion\logs
Maximum file size 5000 KB
Maximum number of archives 10
Log slow pages Yes
Slow page time limit 60 seconds
Log CORBA calls No
Log scheduled tasks No
Schedule Tasks & Probes
Scheduled Tasks
Auto Replenish Notify S9 Order Min Levels Reached
Start Date 9/2/2014
End Date
Interval Daily
Start Time 2:05:00 AM
URL http://server1.example.com/index.cfm?fuseaction=rfid.SendReplenishEmails
Username
Timeout
Proxy Server
Proxy Port 80
Save output to file No
Publish path
Publish filename
Resolve URLs No
Auto Replenish Notify Customer Badge Replenishment
Start Date 9/2/2014
End Date
Interval Daily
Start Time 2:00:00 AM
URL http://server1.example.com/uri1/act_notifyUserBadge.cfm
Username
Timeout
Proxy Server
Proxy Port 80
Save output to file No
Publish path
Publish filename
Resolve URLs No
Auto Replenish
Start Date 9/2/2014
End Date
Interval Daily
Start Time 2:10:00 AM
URL http://server1.example.com/uri1/uri2/uri3/Replenish2.cfm
Username
Timeout
Proxy Server
Proxy Port 80
Save output to file No
Publish path
Publish filename
Resolve URLs No
Contact Updates and Registrations
Start Date 9/2/2014
End Date
Interval 65
Start Time 12:05:00 AM
URL http://server1.example.com/index.cfm?fuseaction=intCRM.sendCRMObjects
Username
Timeout
Proxy Server
Proxy Port 80
Save output to file No
Publish path
Publish filename
Resolve URLs No
System Probes
Extensions
Java Applets
CFX Tags
cfx_xml2cfx
Tag name cfx_xml2cfx
Description
Type CPP
Server library c:\cfusion8\cfx\xml2cfx\cfx_xml2cfx.dll
Keep library loaded true
Procedure ProcessTagRequest
cfx_encrypt_aes
Tag name cfx_encrypt_aes
Description DB encryption library
Type CPP
Server library C:\cfusion8\cfx\aes\cfx_encrypt_aes.dll
Keep library loaded true
Procedure ProcessTagRequest
cfx_exec
Tag name cfx_exec
Description
Type CPP
Server library c:\CFusion8\cfx\exec\cfx_exec.dll
Keep library loaded true
Procedure ProcessTagRequest
Custom Tag Paths
D:\web\app1\app2\app3\activedit3
D:\ColdFusion10\cfusion/CustomTags
D:\ColdFusion8\CustomTags
CORBA
Selected connector [ none]
Connectors
Event Gateways
Settings
Enable Event Gateway Yes
Thread Pool Size 10
Max Queue Size 25000
Gateway Types
SMS
Description Handles SMS text messaging
Class coldfusion.eventgateway.sms.SMSGateway
Timeout 30 seconds
Kill On Timeout Yes
XMPP
Description Handles XMPP instant messaging
Class coldfusion.eventgateway.im.XMPPGateway
Timeout 30 seconds
Kill On Timeout Yes
SAMETIME
Description Handles Lotus SAMETIME instant messaging
Class coldfusion.eventgateway.im.SAMETIMEGateway
Timeout 30 seconds
Kill On Timeout Yes
DirectoryWatcher
Description Watches a directory for file changes
Class examples.watcher.DirectoryWatcherGateway
Timeout 30 seconds
Kill On Timeout Yes
Socket
Description Listens on a socket
Class examples.socket.SocketGateway
Timeout 30 seconds
Kill On Timeout Yes
CFML
Description Handles asynchronous events through CFCs
Class coldfusion.eventgateway.cfml.CfmlGateway
Timeout 30 seconds
Kill On Timeout Yes
JMS
Description Handles Java Messaging Service messages
Class examples.JMS.JMSGateway
Timeout 30 seconds
Kill On Timeout Yes
ActiveMQ
Description Handles Apache ActiveMQ JMS messages
Class examples.ActiveMQ.JMSGateway
Timeout 30 seconds
Kill On Timeout Yes
DataServicesMessaging
Description Handles Data Services Messaging messages
Class coldfusion.eventgateway.flex.FlexMessagingGateway
Timeout 30 seconds
Kill On Timeout No
FMS
Description Handles Flash Media Server shared objects
Class coldfusion.eventgateway.fms.FMSGateway
Timeout 30 seconds
Kill On Timeout Yes
DataManagement
Description Notifies Data Management Services of data changes
Class coldfusion.eventgateway.flex.FlexDataManagementGateway
Timeout 30 seconds
Kill On Timeout Yes
Security
CF Admin Authentication
Enable authentication for the ColdFusion Administrator Yes
Allow access to ColdFusion Administrator with a Single password No
RDS Authentication
Enable authentication for RDS access Yes
Allow access through RDS with Single password Yes
Security Sandboxes
Enable ColdFusion Security No
jvm.config
#
# VM configuration
#
# Where to find JVM, if {java.home}/jre exists then that JVM is used
# if not then it must be the path to the JRE itself
java.home=D:\\ColdFusion10\\jre
application.home=D:\\ColdFusion10\\cfusion
#
# If no java.home is specified a VM is located by looking in these places in this
# order:
#
# 1) ../runtime/jre
# 2) registry (windows only)
# 3) JAVA_HOME env var plus jre (ie $JAVA_HOME/jre)
# 4) java.exe in path
#
# Arguments to VM
java.args=-server -Xms768m -Xmx1280m -XX:MaxPermSize=192m -XX:+UseParallelGC -Xbatch -Dcoldfusion.home={application.home} -Dcoldfusion.rootDir={application.home} -Dcoldfusion.libPath={application.home}/lib -Dorg.apache.coyote.USE_CUSTOM_STATUS_MSG_IN_HEADER=true -Dcoldfusion.jsafe.defaultalgo=FIPS186Random -Dges.config.serverEnvironment=prod2 -Dges.config.webServerHostName=server1.example.com -Dcom.sun.management.jmxremote.port=8880 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcoldfusion.classPath={application.home}/lib/updates,{application.home}/lib,{application.home}/lib/axis2,{application.home}/gateway/lib/,{application.home}/wwwroot/WEB-INF/flex/jars,{application.home}/wwwroot/WEB-INF/cfform/jars
# Comma separated list of shared library path
java.library.path={application.home}/lib,{application.home}/jintegra/bin,{application.home}/jintegra/bin/international,{application.home}/lib/oosdk/classes/win
java.class.path={application.home}/lib/oosdk/lib,{application.home}/lib/oosdk/classes
Copy link to clipboard
Copied
How many active sessions are there when you start to have the OOM and blocked threads issues? Your app may be holding on to sessions longer than it really needs to. If you get a lot of traffic and bots, store a lot of info in the session scope and they don't time out fast enough, you will have OOM and blocked threads issues.
I had this happen earlier this year and it was a nightmare to figure out that this was the problem.
Copy link to clipboard
Copied
When I go to the server monitor and view active sessions it's showing anywhere from 4-5K active sessions when the blocked threads are not present. I'll have to check keep checking regularly and see if I can find a trend relating to the blocked threads. Note that in the thread dump we've found two ajp threads blocked on each other so it's a complete deadlock which it appears it'll never recover from.
Copy link to clipboard
Copied
4-5K active sessions isn't a lot but I'd like to see if that number keeps growing or levels out.
Are you running just the one cfusion instance?
What is the sessionTimeout set to in each applcation.cfc?
What are you putting in your client and session scopes?
Are you seeing high CPU usage when you start seeing the OOM errors?
It looks like your using a database to store your client variables but "Purge data" isn't checked. If I understand correctly, you should have it set to purge data. Otherwise the client tables in the database will grow out of control and cause some real performance issues. I've never used client vars with a database so I could be very wrong.
Copy link to clipboard
Copied
After running for about 48 hours the number of active sessions doesn't grow much (about 500). We're running 2 CF 10 instances on separate VMs. Traffic goes through 2 apache servers with each apache pointed directly at one CF instance, no load balancing at the app level. It looks like the session timeout is set to 45 minutes, but I'm not sure. Seeing the below line in the code.
THIS.SessionTimeout = "#CreateTimeSpan(0,0,45,0)#";
I'm not a developer so I don't know the answer to what's in the client and session scopes. I'll have to see if I can find a dev who can answer that. We do see CPU usage gradually climb due to the amount of time being spent in GC as the heap gets full. When we finally get an OOM error we sometimes see very high CPU or just above normal. I also verified the Purge Data option is checked in the console and it's set to 90 days.
*EDIT* Looking at the client variables section in more detail I see our default option is a database. It currently does not purge the data and the box for "" is checked.
Copy link to clipboard
Copied
Definitely expiring sessions in 45 min so that sounds fine.
In the CF Admin Server Monitor you can can go to the "Statistics" tab and open "Memory Usage" then click "Memory Usage /summary". How much memory is being used by sessions, application, and server? Can you post a screenshot of this page so I can see what the garbage collection graph and memory usage graph look like?
Its common to see CPU spikes when GC runs. Usually with memory issues it will get to the point where GC is running longer than it should and the CPU will eventually max out.
At this point I don't think its anything to do with sessions but I want to see what is going on to be on the safe side. It sounds more like an issue with the way client vars are being used and I don't have a lot of experience with them. It is good to know you don't have "" checked. For the most part it adds overhead to each page that isn't needed.
Purge Data is on so that is good but 90 days is a long time to store client vars. You timeout sessions after 45 min. Why keep client vars around for 90 days? Unless there is a good reason to keep client vars that long, I would reduce the Purge Data to something like 2-3 days. My reasoning. Every time someone comes to your site you are creating a new session to store in server memory and new set of client vars to store in a database. The sessions are removed from memory after they time out. Right now your client vars are being stored and saved for ever in your database in tables that most likely are not indexed. This will cause performance issues as the tables fill up. According to the docs, "To improve performance, create indexes when you create these tables. For the CDATA table, index these cfid and app columns. For the CGLOBAL table, index the cfid column." I would also use a dedicated database for client vars.
I looked over your JVM arguments and have some suggestions.
Its recommended to have Xms and Xmx and -XX:PermSize and -XX:MaxPermSize set to the same value. I would consider changing your settings to -Xms1280m -Xmx1280m -XX:PermSize=192m -XX:MaxPermSize=192m. If you have enough server memory you may want to allocate more to Xms and Xmx.