Highlighted

Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?

Participant ,
Jul 22, 2015

Copy link to clipboard

Copied

We have a development copy of CF11 that we installed as a RTM build that has since gone through CU1 through CU5.  We just recently installed CF11 on our production server which was done from the latest deployment package and was then upgraded to CU4 then CU5.  We've noticed instabilities with the production copy that we haven't seen once on the development copy the years it has been running.

The most common are "Error Executing Database Query. Timed out trying to establish connection" errors at random while other applications using the same DB server have no issues at all.  If you reload the page the DB call goes through the 2nd time.  Only CF applications on this web server have these issues and not .NET or Java based ones.

So wondering if I had configuration wrong I merge compared our development versus production server and found a number of core parts of the CF11 that started as RTM to be much older.  Various jar files, the Jetty runtime is v6 instead of v9, tomcat files, etc.  It's obvious at some point the CF11 deployment package has been updated but typically in the past we'd see a CF 9.0.1 type of release for major deployment changes but both of my servers report as being Version 11,0,05,293506.

After countless attempts to stop this from happening the solution we finally found was to run the Microsoft JDBC drivers v4.2 inside of ColdFusion11. Once we stopped using all Adobe JDBC drivers to connect to our MS-SQL 2014 server the server stopped being unstable. If even one active DB is using the built in Adobe MS-SQL driver then the system in both our development and production enviroment will eventually behave as documented previously.

https://www.microsoft.com/en-us/download/details.aspx?id=11774

  1. CF11 CFIDE -> Data Sources -> Type name as usual but choose “other” as driver
  2. JDBC URL is where you define the connection string including the server to connect to, the DB, and encryption (if you use encryption make sure to use the FQDN
    1. jdbc:sqlserver://SERVERNAME:PORT;databaseName=DATABASETOUSE;encrypt=TRUE;
  3. Driver Class must contain the name of the JDBC driver
    1. com.microsoft.sqlserver.jdbc.SQLServerDriver
  4. Driver Name must contain the correct name
    1. SQLServerDriver
  5. Fill out the User Name and Password as usual
  6. The Advanced area contains all of the same settings except for the “Max Pooled Statements”

I have a set of benchmark scripts that run an exhaustive set of DB queries I use to evaluate performance changes over time.  On my development test server these take 630 seconds to run on average and moving to the MS JDBC driver the on average time dropped by 30 seconds to complete for unencrypted connections.  For encrypted connections it took on average the same amount of time to complete.

One issue we’ve discovered is that the driver enforces some MS coding practices by disallowing the running of certain queries. For example a select identity inside an insert query will error using the MS JDBC driver but run with the Adobe driver.

Views

911

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Why is CF11 RTM + CU5 drastically different from the latest deployment package + CU5?

Participant ,
Jul 22, 2015

Copy link to clipboard

Copied

We have a development copy of CF11 that we installed as a RTM build that has since gone through CU1 through CU5.  We just recently installed CF11 on our production server which was done from the latest deployment package and was then upgraded to CU4 then CU5.  We've noticed instabilities with the production copy that we haven't seen once on the development copy the years it has been running.

The most common are "Error Executing Database Query. Timed out trying to establish connection" errors at random while other applications using the same DB server have no issues at all.  If you reload the page the DB call goes through the 2nd time.  Only CF applications on this web server have these issues and not .NET or Java based ones.

So wondering if I had configuration wrong I merge compared our development versus production server and found a number of core parts of the CF11 that started as RTM to be much older.  Various jar files, the Jetty runtime is v6 instead of v9, tomcat files, etc.  It's obvious at some point the CF11 deployment package has been updated but typically in the past we'd see a CF 9.0.1 type of release for major deployment changes but both of my servers report as being Version 11,0,05,293506.

After countless attempts to stop this from happening the solution we finally found was to run the Microsoft JDBC drivers v4.2 inside of ColdFusion11. Once we stopped using all Adobe JDBC drivers to connect to our MS-SQL 2014 server the server stopped being unstable. If even one active DB is using the built in Adobe MS-SQL driver then the system in both our development and production enviroment will eventually behave as documented previously.

https://www.microsoft.com/en-us/download/details.aspx?id=11774

  1. CF11 CFIDE -> Data Sources -> Type name as usual but choose “other” as driver
  2. JDBC URL is where you define the connection string including the server to connect to, the DB, and encryption (if you use encryption make sure to use the FQDN
    1. jdbc:sqlserver://SERVERNAME:PORT;databaseName=DATABASETOUSE;encrypt=TRUE;
  3. Driver Class must contain the name of the JDBC driver
    1. com.microsoft.sqlserver.jdbc.SQLServerDriver
  4. Driver Name must contain the correct name
    1. SQLServerDriver
  5. Fill out the User Name and Password as usual
  6. The Advanced area contains all of the same settings except for the “Max Pooled Statements”

I have a set of benchmark scripts that run an exhaustive set of DB queries I use to evaluate performance changes over time.  On my development test server these take 630 seconds to run on average and moving to the MS JDBC driver the on average time dropped by 30 seconds to complete for unencrypted connections.  For encrypted connections it took on average the same amount of time to complete.

One issue we’ve discovered is that the driver enforces some MS coding practices by disallowing the running of certain queries. For example a select identity inside an insert query will error using the MS JDBC driver but run with the Adobe driver.

Views

912

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Jul 22, 2015 0
Guide ,
Jul 22, 2015

Copy link to clipboard

Copied

Hi Brent,

The CF11 release installer laid down Java 7. The latter CF11 installer uses Java 8. If one went release then U1 to U5 that would not alter the underlying Java version.

I have seen issues where database connections are broken due to a poorly performing Java heap or non heap issue. You could do well to check how well the heap or non heap IE metaspace values are performing and if one or other need adjustment. Not to be forgotten with Java memory is CodeCache and CompressedClass to know either of those are consuming memory in good boundaries. Use traditional Java logging to a file or enable JMX then use JDK tools like jconsole or jmc to check the memory consumption is stressed or not.

HTH, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 22, 2015 0
Participant ,
Jul 23, 2015

Copy link to clipboard

Copied

I noticed our development server still had -XX:MaxPermSize=192m where the new production server had -XX:MaxMetaspaceSize=192m set instead.  Which reading up on that setting not having it defined would have given an unlimited access size.  I've changed the development server to limit to 192m to see if it causes issues there.  Are there any good articles written on metaspace tuning for CF like the connector one?

We don't run the internal Java but instead external JRE copies so both machines are running JRE 1.8.0_45 (working on upgrading to _51 soon).

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 23, 2015 0
Adobe Employee ,
Jul 23, 2015

Copy link to clipboard

Copied

 

Regarding the Performance issue, here are some resources, that will help you.

 

* Performance tuning for ColdFusion applications (http://www.adobe.com/devnet/coldfusion/articles/coldfusion_performance.html),

*         JVM Tuning (http://docs.oracle.com/cd/E13222_01/wls/docs81/perform/JVMTuning.html)

 

These articles are for JRE 7, however, the logic remains the same for JRE 8 as well.

 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 23, 2015 0
Participant ,
Jul 23, 2015

Copy link to clipboard

Copied

After setting my development system to have the same -XX:MaxMetaspaceSize=192m I started getting the same errors just browsing around a development website.

Error Executing Database Query. 

  Timed out trying to establish connection 

...

java.sql.SQLException: Timed out trying to establish connection

    at coldfusion.server.j2ee.sql.pool.JDBCPool.requestConnection(JDBCPool.java:843)

    at coldfusion.server.j2ee.sql.pool.JDBCManager.requestConnection(JDBCManager.java:125)

    at coldfusion.server.j2ee.sql.JRunDataSource.getConnection(JRunDataSource.java:137)

    at coldfusion.sql.CFDataSource.getConnection(CFDataSource.java:44)

    at coldfusion.sql.DataSrcImpl.getCachedConnection(DataSrcImpl.java:156)

    at coldfusion.sql.DataSrcImpl.getConnection(DataSrcImpl.java:110)

    at coldfusion.sql.SqlImpl.execute(SqlImpl.java:363)

    ...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 23, 2015 0
Guide ,
Jul 23, 2015

Copy link to clipboard

Copied

Seems to me you are creeping up on the answer since setting dev to -XX:MaxMetaspaceSize=192m 

you get the database connectivity problem. So you are consuming  greater than 192m of MetaSpace which

is not unusual on 64bit.

With Java 8 it is able to manage MetaSpace automatically if you do not define a value for MaxMetaspaceSize= however it may not make good decisions and will be inclined to re-size down to it's lower value. All this resizing invokes a Full garbage collection which is a performance loss.

You can do well to define an initial and maximum setting for MetaSpace. I find this works well in most cases:

-XX:MetaspaceSize=312m -XX:MaxMetaspaceSize=512m

Having said that some Java logging or JMX monitoring might reveal better values to apply given the

type of workload.

HTH again, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 23, 2015 0
Participant ,
Jul 24, 2015

Copy link to clipboard

Copied

I enabled JMX and installed VisualVM so I've been able to watch the resource usage.  Oddly the DB timeout happened when only 80mb of metaspace was in use when limited to 192.  We were doing similar min/max settings with the previous Java 7 settings, thank you for providing me the terms for the new settings.

I'm also looking into tomcat connector tuning as well but I don't think the DB timeouts are related to it (at least I don't think they are).

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 24, 2015 0
Guide ,
Jul 26, 2015

Copy link to clipboard

Copied

It is a good idea to apply tomcat tuning tho like you say adjusting tomcat pools, threads and timeout is probably not related to the database timeout problem.

I guess you have adjusted MetaSpace. Does Jvisualvm show it consuming more memory now?

While you have JMX enabled you might like use jconsole or jmc to check CodeCache and CompressClass memory consumption to know those values are well.

Regards, Carl.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Jul 26, 2015 0
Participant ,
Aug 18, 2015

Copy link to clipboard

Copied

Unfortunately the increased memory space and Tomcat tuning did not resolve the issue on our production servers.  I was hoping they might happen less frequently but they still seem to happen at about the same rate. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 18, 2015 0
Participant ,
Aug 18, 2015

Copy link to clipboard

Copied

I've created a Adobe Bug for the timeouts as we've been unable to make them stop on our production environment and can reproduce them on development as well.

https://bugbase.adobe.com/index.cfm?event=bug&id=4038865

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 18, 2015 0
Participant ,
Sep 02, 2015

Copy link to clipboard

Copied

After countless attempts to stop this from happening the solution we finally found was to run the Microsoft JDBC drivers v4.2 inside of ColdFusion11. Once we stopped using all Adobe JDBC drivers to connect to our MS-SQL 2014 server the server stopped being unstable. If even one active DB is using the built in Adobe MS-SQL driver then the system in both our development and production enviroment will eventually behave as documented previously.

https://www.microsoft.com/en-us/download/details.aspx?id=11774

  1. CF11 CFIDE -> Data Sources -> Type name as usual but choose “other” as driver
  2. JDBC URL is where you define the connection string including the server to connect to, the DB, and encryption (if you use encryption make sure to use the FQDN
    1. jdbc:sqlserver://SERVERNAME:PORT;databaseName=DATABASETOUSE;encrypt=TRUE;
  3. Driver Class must contain the name of the JDBC driver
    1. com.microsoft.sqlserver.jdbc.SQLServerDriver
  4. Driver Name must contain the correct name
    1. SQLServerDriver
  5. Fill out the User Name and Password as usual
  6. The Advanced area contains all of the same settings except for the “Max Pooled Statements”

I have a set of benchmark scripts that run an exhaustive set of DB queries I use to evaluate performance changes over time.  On my development test server these take 630 seconds to run on average and moving to the MS JDBC driver the on average time dropped by 30 seconds to complete for unencrypted connections.  For encrypted connections it took on average the same amount of time to complete.

One issue we’ve discovered is that the driver enforces some MS coding practices by disallowing the running of certain queries. For example a select identity inside an insert query will error using the MS JDBC driver but run with the Adobe driver.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Sep 02, 2015 0