Webservice call results in Java heap space error

[chris hawk_448]

Migrating to CF2018, we have noticed that our webservice calls are quickly pegging system resources and resulting in java heap space out of memory errors after a few minutes.  Increasing memory allocation results in longer periods of time before erroring out, but similar results.

Pointing at java 8 and moving tools.jar into the WEB-INF/lib directory shows successful results with near instantaneous response.

Has anyone else seen this or have any idea why java 10 would be showing this issue?

[chris hawk_448]

Here is the code for a simple call that is failing for us in the same way:

Hello.cfc

<cfcomponent displayname="Hello">

  <cffunction name="getHelloWorld" access="remote" returntype="string" output="no">

    <cfreturn "Received hello from secure web service">

  </cffunction>

</cfcomponent>

HelloClient.cfm

<cftry>

  <cfinvoke webservice ="#variables.baseURL#/ATS-UI/Hello.cfc?wsdl"

    method ="getHelloWorld"

    username="#####"

    password="#####"

    returnVariable="greeting">

  </cfinvoke>

  <cfdump var="#greeting#">

  <cfcatch type="Any">

    <h2 class="importanttext">Error:</h2>

    <cfoutput>

      <p>#cfcatch.errorCode# - #cfcatch.message#</p>

      <p>#cfcatch.detail#</p>

      <p>Additional details:</p>

      <cfset ExceptionMsg = "">

      <cfloop index = i from = 1

        to = #ArrayLen(CFCATCH.TAGCONTEXT)#>

        <cfset sCurrent = #CFCATCH.TAGCONTEXT#>

        <CFSET tmpMsg = "Line, Col: (#sCurrent['LINE']#,#sCurrent['COLUMN']#)<br>

        Template:   #sCurrent['TEMPLATE']#">

        <cfset ExceptionMsg = "#VARIABLES.ExceptionMsg# #chr(13)#" & "#variables.tmpMsg#">

      </cfloop>

    </cfoutput>

  </cfcatch>

</cftry>

[BKBK]

Hi https://forums.adobe.com/people/chris+hawk_448 , thanks for the test code.

There were no problems when I ran it. My system is ColdFusion 2018 + Java 10.0.2. The likely cause of your problem is therefore something else in your environment, something outside the code,.

I suspect the webservice version. That is, the Axis version. Mine is 2 whereas yours is 1.

Go to the webservice page in the ColdFusion Administrator. Select 2 as the web service version, then click on the button to update it.

To be sure add two more attributes, wsversion and refreshwsdl, to your cfinvoke tag:

<cfinvoke webservice ="#variables.baseURL#/ATS-UI/Hello.cfc?wsdl"

    method ="getHelloWorld"

    username="#####"

    password="#####"

    returnVariable="greeting" wsversion="2" refreshwsdl="true" >

You may remove the refreshwsdl later, after you get the service to work.

[Charlie Arehart]

A few thoughts for you both.

Bkbk, I had proposed that (refresh of the wsdl, any of 3 ways) in my last comment, and Chris said it had not helped. But I'm with you in saying this feels like an environmental thing for him.

So Chris, first, had you applied any of the updates to your cf2018? If so, have you confirmed there were 0 errors, in the table tracking successes and errors near the top of the install log for the update, in the cfusion/hf-updates folder? That's always important to check after an update.

Second, I had asked on the 21st if you had any monitoring of heap use, like fr or the pmt. My goal then was to see if you see heap use rising all day.

But now I'll ask: if you DO have the PMT implemented and this instance is being monitored by it, try disabling that (in the PMT), then run your tests. (And do restart CF if you found heap had risen all day and was now high, so that you have a clear starting point of comparison with the PMT monitoring now disabled.)

My point here is that I have seen at least one problem caused BY the the fact that a CF instance was being monitored by the PMT. (it was related to errors in cfthread  so not related to this) .

Still, the PMT DOES in fact monitor all calls OUT of CF--whether db calls, cfhttp calls, web svc calls, and more--so it COULD be somehow mistakenly causing CF to hold on to objects related to the ws call, in effect causing a memory leak.

So taking it out of the equation would be worth a shot if you do have it enabled for this instance. If not--and if you confirm there were also no errors in the update install log--then I think I'm out of ideas for now. 🙂

[BKBK]

https://forums.adobe.com/people/Charlie+Arehart  wrote

A few thoughts for you both.

Bkbk, I had proposed that (refresh of the wsdl, any of 3 ways) in my last comment, and Chris said it had not helped.

Hi Charlie, my suggestion is for Chris to change the webservice version from 1 to 2.

[chris hawk_448]

I have tried with wsversion=2 and refreshwsdl=true.  Neither seemed to have an effect.

I'm currently on update level 2 and don't see any errors in the install logs.

I have been running with the PMT disabled in order to not take up any further resources on the server.  When I did have it enabled, I could see the heap rising steadily after a WS call.  It looked fine prior to that time, however.

I'm going to go back and look over my environment again, maybe try a different version of java or try this out on a new machine.  This is continuing to hold me up from moving forward on our upgrade, so I need to find some answers.  I appreciate all of the ideas so far, though!

[BKBK]

https://forums.adobe.com/people/chris+hawk_448  wrote

I have tried with wsversion=2 and refreshwsdl=true.  Neither seemed to have an effect.

I'm currently on update level 2 and don't see any errors in the install logs.

To avoid any possible Update 1 hiccups, I would uninstall ColdFusion 2018. I would then download and install, using the latest ColdFusion 2018 installer, which is Update 2.

[chris hawk_448]

I will give that a try and get back to you.

[chris hawk_448 · Accepted solution]

Update:

I uninstalled CF2018, reinstalled using the current installer with update 2, and then began to test using a minimally configured setup (mostly default settings).  After quite a bit of debugging, I believe I have narrowed down the problem to two things:

1.  It appears that we have a bug in our code that has been known to cause problems consuming CF published webservices.  This is outlined in the following doc, but curiously wasn't an issue for us using java 8...

https://tracker.adobe.com/#/view/CF-3581691

2.  We have been using the javax.servlet jar file for some time now.  With java 10, this seems to be causing our webservice calls to being running up the CPU and increasing java heap until a heap error occurs.  Removing this jar fixes the problem.  I have yet to determine if an updated jar is necessary.

[BKBK]

https://forums.adobe.com/people/chris+hawk_448  wrote

Update:

I uninstalled CF2018, reinstalled using the current installer with update 2, and then began to test using a minimally configured setup (mostly default settings).  After quite a bit of debugging, I believe I have narrowed down the problem to two things:

1.  It appears that we have a bug in our code that has been known to cause problems consuming CF published webservices.  This is outlined in the following doc, but curiously wasn't an issue for us using java 8...

https://tracker.adobe.com/#/view/CF-3581691

That bug dates from 2013 and ColdFusion 10. That's quite old in software terms. The Adobe ColdFusion Team would have sorted the issue out by now.

2.  We have been using the javax.servlet jar file for some time now.  With java 10, this seems to be causing our webservice calls to being running up the CPU and increasing java heap until a heap error occurs.  Removing this jar fixes the problem.  I have yet to determine if an updated jar is necessary.

A separate javax.servlet jar in ColdFusion 2018? That is unnecessary!

Remove the additional JAR for a moment. Restart ColdFusion 2018. Run the following code (without the additional JAR):

<cfset servletObject=createobject("java","javax.servlet.Servlet")>

<cfdump var="#servletObject#">

You will find that it works. This proves that ColdFusion 2018 contains the javax.servlet package.

[BKBK]

-Xms256m -Xmx512m

That may be too small for the work your application is doing. It certainly is too small for the G1GC collector. With G1GC I would recommend values of at least

-Xms4096m -Xmx4096m

This assumes you have sufficient RAM. If you haven't, then change the collector to

-XX:+UseParallelGC

and use the value you mentioned before:

-Xms2048m -Xmx2048m

Lastly, you miss a flag which ColdFusion 2018 adds:

-Dcom.sun.xml.bind.v2.bytecode.ClassTailor.noOptimize=true

It improves performance.