Participant

Question

Empty response from server

Forum|Forum|7 years ago
October 18, 2018
6 replies
6645 views

I have a script that, when called from a web browser, returns an empty response. consider the following code:

-------------------------------------------------------------------

hello

</cfstoredproc>

<cfoutput>#result.OkToContinue#</cfoutput>

goodbye

-------------------------------------------------------------------

If I move the "goodbye <cfexit>" above the call to the stored procedure, it displays "hello goodbye" and quits successfully. If I leave the <cfexit> below the <cfstoredproc> call, it returns an empty response from the server. It doesn't display hello, or goodbye at all. I don't think it's timing out because it returns an empty response very quickly.

ColdFusion 2018 Enterprise

CentOS 7 environment

I'm sure it has something to do with my ColdFusion configuration because this exact same script runs without any issue in our Windows environment. Any help would be greatly appreciated.

Thanks in advance,

Rick

Server administration

This topic has been closed for replies.

Charlie Arehart

Community Expert

Rob, could you update us on the state of things for you? And had you considered my last comment?

And one other, for anyone else finding this: do make sure you have UPDATED the connector (running the wsconfig tool to do an update, which puts in place a new mod_sk.so) In CF2018, update 2 had introduced an updated connector.

/Charlie (troubleshooter, carehart. org)

R

rob2019

Participant

Hi Charlie.

Thank you for your replies earlier this year. My apologies for not responding earlier however the forum failed to email me that there was a reply and so I didn't see your comments until this week (November 2019).

AFAICT we've been stable on update 4 using Apache 2.4 with MPM Event as I described in my earlier post.

Installing update 5 introduces instability and after approximately 8 to 24 hours our servers with update 5 end up in some kind of error state where all page loads return server timout error messages.

We are not using PMT - however we also never manually recreated the connector (mod_jk). I had no idea this was required - always assumed this was handled by the coldfusion update process that one initiates in the CFIDE.

So we are still using the mod_jk.so that came with ColdFusion 2018.

I ran

wsconfig -upgrade -v

and I found a new mod_jk.so in

/etc/httpd/conf

I was expecting it to be installed in

coldfusion2018/config/wsconfig/1/

or thereabouts.

I have not been able to find any official ColdFusion documenation that walks you through this process. I assume at this point I need to manually replace the mod_jk.so in "coldfusion2018/config/wsconfig/1/" with the new one in "/etc/httpd/conf" - and restart Apache & ColdFusion?

W

Wiolant

Participant

In the meanwhile we are quite sure that it is a problem with the connector. We exchanged the mod_jk.so in /opt/coldfusion2018/config/wsconfig/1/ with the mod_jk.so from ColdFusion 2016. Now it works.

M

Marco_Mueller

Participant

Change of

heartbeat_interval=30 to heartbeat_interval=0

in the file [apache_home]/workers.properties

eg. /etc/apache2/workers.propertiers on SuSE

solved the problem of blank pages in ColdFusion2018 for us (and a lot of more problems as well like eg. cfflush....)

I got this info from the Adobe Support Team and they told me this should be solved in the next update.

This entry is new in the workers.properties and is used by the Performance Monitor.

Performance Monitor should be able to run as well with the modified Setting.

Changed as well the [cf_home]/cfusion/bin/jvm.config

to use G1GC instead of parallelGC

-XX:+UseParallelGC changed

to

-XX:+UseG1GC

see for documentation below link

https://www.oracle.com/technetwork/tutorials/tutorials-1876574.html#t2

You need a restart of ColdFusion and Apache for the changes to take effect.

Charlie Arehart

Community Expert

That's indeed interesting to hear, Marco (that setting the heartbeat_interval to 0 solved your problem). I would surely hope we hear more from Adobe about that, before everyone seeing this may go change that. What does it even do?

As you say, it's about the PMT. We need to know what exactly happens when that's set to 0?

There's scant doc on the heartbeat interval. Indeed, about all I could easily find was this from the CF2018 what's new docs (https://helpx.adobe.com/coldfusion/using/whats-new.html), which say that it is "The time interval in seconds after which the connector load data is sent to ColdFusion."

Does it matter what it is if one does NOT have the PMT enabled? What if that instance in question is not being monitored by the PMT?

Also, you end by mentioning a change to using the G1GC. That's interesting, but it would help to know if that had ANYTHING to do with your resolution. I would expect not. Again, I fear some may see this and having that problem think, "oh, let's change our GC also", which could have unintended consequences. Always best to change one thing at a time, when trying to solve a problem. :-)

Anyway, let's see if Adobe may offer more (to you or here). Thanks for sharing what you heard.

/Charlie (troubleshooter, carehart. org)

W

Wiolant

Participant

Did you manage to solve this issue? We are using ColdFusion 2018 Standard (Update 1) together with SLES12 SP3 and face quiet the same problem...

rob34950395

Participant

We have encountered this issue with CF18 while using Apache 2.4.x prefork on RedHat 7. I'm sharing my experience here to document it and in hopes it may help others.

To summarize:

mod_jk is crashing, specifically function: jk_connector_CF_func
it looks like I was able to fix this (still testing to confirm) by using MPM Event instead of MPM Prefork

Details:

While setting up severs to run CF18, over the course of a couple months a few times I noticed an ERR_EMPTY_RESPONSE type error message but didn't think much of it. As we got closer to bringing this system live - I started to test longer running URLs and the issue became quite clear.

From my testing, I found that pages that took >20 seconds to complete processing had a high likelihood (>50%) that the connection would be dropped - thus producing an ERR_EMPTY_RESPONSE. Page processing greater than 30 seconds had a likelihood close to 100%.

Using a long running page to test this issues showed dropped connections anywhere from a few seconds out to 20 or more - but again - by 30 seconds elapsed time there was almost always guarantee of dropped connection and thus generation of ERR_EMPTY_REPONSE.

Adding ouptut and flush statements to the code so that the server would send responses back to the browser on a regular (<=5 seconds) interval had no effect on the dropped connections - however the error changed to ERR_INCOMPLETE_CHUNKED_ENCODING.

Ajax calls receiving ERR_EMPTY_RESPONSE would repeat the call one additional time - producing double processing log entries. Once the error changed to ERR_INCOMPLETE_CHUNKED_ENCODING Chrome AJAX calls would issue an error while Firefox AJAX calls stated status was OK (using jquery library). Regardless - the underlying connection issue was unsolved and still an issue.

To eliminate any further double calling - started using curl to test requests going forward.

Initial review of Apache log files showed nothing - literally - as anytime a connection was dropped - there would be no corresponding entry in Apache for that request (even though it was processed by Coldfusion). We checked separate backend logs and verified that the requests were coming through to Coldfusion and completing - but for some reason the connection from Apache to the browser was dropping.

Eventually I found, as others have noted above, a warning in mod_jk logs that seemed strange:

[Thu Feb 28 20:50:00 2019] [9137:139898250012416] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

The timing of the error was correlated to URL requests where we experienced a dropped connection.

Turning up debug/trace type logging in Apache led me to this:

[Thu Feb 28 19:14:23.317994 2019] [core:notice] [pid 7212] AH00052: child pid 7371 exit signal Segmentation fault (11)

Next fire up the GDB debbuger in linux, limit Apache to one process, attach to the Apache process that receives requests, running a CURL test and once it drops connection I see the following output:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f9778ff9700 (LWP 7439)]
0x00007f9785f0aeb4 in jk_connector_CF_func (thd=<optimized out>, data=<optimized out>) at mod_jk.c:4143
4143 mod_jk.c: No such file or directory.

So at this point looks like mod_jk is crashing and because of some issue with the jk_connector_CF_func which clearly looks like it was added by Adobe for integration with Coldfusion (looking at the publicly available source code at line 4143 brings you to the end of the file).

I am still testing this - but it looks like I was able to fix this issue by using MPM Event in Apache - as others here have noted. I am using the following configuration which does not generate any warnings by Apache when checking via configtest:

<IfModule mpm_event_module>
        StartServers            2
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit              500
        ThreadsPerChild          500
        MaxRequestWorkers         500
        MaxConnectionsPerChild   0
</IfModule>

I will still be conducting more testing this week - and will update if there are any other significant changes or findings.

rob34950395

Participant

confirming that through my testing AFAICT the recent CF update (3) from around March 4 2019 does NOT fix this issue.

After applying ColdFusion (2018 release) Update 3 , Coldfusion version now shows as: 2018,0,03,314033

Post applying update 3 testing on servers running Apache 2.4.x...

configured to use MPM prefork continue to fail with either ERR_EMPTY_RESPONSE or ERR_INCOMPLETE_CHUNKED_ENCODING
configured to use MPM event continue to succeed in my tests so far

BKBK

Community Expert

ricks29770624 wrote
I have a script that, when called from a web browser, returns an empty response. consider the following code:
-------------------------------------------------------------------
hello
    <cfstoredproc procedure = "ParseAPStoriesExecDTSCentOS" datasource = "#app_datasourceAdminUser#" >
            <cfprocparam type ="OUT" cfsqltype="CF_SQL_INTEGER" variable=OkToContinue dbvarname="@OkToContinue">
                <cfprocresult name="result">
    </cfstoredproc>
    <cfoutput>#result.OkToContinue#</cfoutput>
goodbye
<cfexit>

OK. For the sake of curiosity, what happens when you place the following line before 'hello'?

R

ricks29770624Author

Participant

I haven't gone back and looked at this since the script was fixed. What was it that you suspect was happening? What is cfflush supposed to do here?

BKBK

Community Expert

I suspect a thread is hanging. Cfflush is supposed to flush some letters to the client.

James Trujillo_466

Participating Frequently

We are getting a very similar issue... we testing Ubuntu/Apache with CF18 with hopes for deployment and we run into the empty_response issue quite frequently. However, at this time we have not nailed down any one piece of script that is causing it. When we run the SAME application on Windows/IIS with CF18, the error NEVER gets triggered. I believe there is an issue with the way unix-based servers are compiling CF18. Granted, I have no pure evidence in this and hope that Adobe does their due diligence to help us figure this out.

D

Dave Watts

Community Expert

I think it's more likely an issue with the web server connector than with the compilation process.

Dave Watts, Fig Leaf Software

Dave Watts, Eidolon LLC

James Trujillo_466

Participating Frequently

You very well might be correct. What worries me is that the unix-based platforms are the ones experiencing this issue. I have just confirmed that we have the same issue on MacOS as well... and again, not on Windows. I will look at the connector to see how I might be able to adjust it, if at all... this is where I might be out of my skill set. If you have any suggestions, I am always open. I will follow up as I make progress.

D

Dave Watts

Community Expert

Out of curiosity, why are you using CFEXIT here? If your script ends, you don't need CFEXIT to explicitly tell CF to end execution. Just for fun, I'd remove that and see what happens.

Dave Watts, Fig Leaf Software

Dave Watts, Eidolon LLC

R

ricks29770624Author

Participant

There is more to the script, I was just using the CFEXIT to determine where the script was dying.

WolfShade

Legend

Have you used CFTRY/CFCATCH on it? Dump the #cfcatch# to either the screen or an email.

V/r,

^ _ ^

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded