Empty response from server

New Here ,
Oct 18, 2018 Oct 18, 2018

Copy link to clipboard

Copied

I have a script that, when called from a web browser, returns an empty response.  consider the following code:

-------------------------------------------------------------------

hello

    <cfstoredproc procedure = "ParseAPStoriesExecDTSCentOS" datasource = "#app_datasourceAdminUser#" >

            <cfprocparam type ="OUT" cfsqltype="CF_SQL_INTEGER" variable=OkToContinue dbvarname="@OkToContinue">

                <cfprocresult name="result">

    </cfstoredproc>

    <cfoutput>#result.OkToContinue#</cfoutput>

goodbye

<cfexit>

-------------------------------------------------------------------

If I move the "goodbye <cfexit>" above the call to the stored procedure, it displays "hello goodbye" and quits successfully.  If I leave the <cfexit> below the <cfstoredproc> call, it returns an empty response from the server.  It doesn't display hello, or goodbye at all.  I don't think it's timing out because it returns an empty response very quickly.

ColdFusion 2018 Enterprise

CentOS 7 environment

I'm sure it has something to do with my ColdFusion configuration because this exact same script runs without any issue in our Windows environment.  Any help would be greatly appreciated.

Thanks in advance,

Rick

TOPICS
Server administration

Views

3.7K

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Oct 18, 2018 Oct 18, 2018

Copy link to clipboard

Copied

Out of curiosity, why are you using CFEXIT here? If your script ends, you don't need CFEXIT to explicitly tell CF to end execution. Just for fun, I'd remove that and see what happens.

Dave Watts, Fig Leaf Software

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Oct 18, 2018 Oct 18, 2018

Copy link to clipboard

Copied

There is more to the script, I was just using the CFEXIT to determine where the script was dying. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Oct 18, 2018 Oct 18, 2018

Copy link to clipboard

Copied

Have you used CFTRY/CFCATCH on it?  Dump the #cfcatch# to either the screen or an email.

V/r,

^ _ ^

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 05, 2018 Nov 05, 2018

Copy link to clipboard

Copied

We are getting a very similar issue... we testing Ubuntu/Apache with CF18 with hopes for deployment and we run into the empty_response issue quite frequently. However, at this time we have not nailed down any one piece of script that is causing it. When we run the SAME application on Windows/IIS with CF18, the error NEVER gets triggered. I believe there is an issue with the way unix-based servers are compiling CF18. Granted, I have no pure evidence in this and hope that Adobe does their due diligence to help us figure this out.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 05, 2018 Nov 05, 2018

Copy link to clipboard

Copied

I think it's more likely an issue with the web server connector than with the compilation process.

Dave Watts, Fig Leaf Software

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 05, 2018 Nov 05, 2018

Copy link to clipboard

Copied

You very well might be correct. What worries me is that the unix-based platforms are the ones experiencing this issue. I have just confirmed that we have the same issue on MacOS as well... and again, not on Windows. I will look at the connector to see how I might be able to adjust it, if at all... this is where I might be out of my skill set. If you have any suggestions, I am always open. I will follow up as I make progress. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Nov 06, 2018 Nov 06, 2018

Copy link to clipboard

Copied

Well, I'm sorry you're having the issue, but I'm glad I'm not the only one. 

I was eventually able to debug the script, but had to do so completely blindly because I wasn't getting any feedback from the CF engine. 

I'll be curious to know what you find.  Definitely keep me posted. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 06, 2018 Nov 06, 2018

Copy link to clipboard

Copied

We have just started the painful process of debugging - blindly as you say. Was there any specific part of your script that was the culprit?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Nov 06, 2018 Nov 06, 2018

Copy link to clipboard

Copied

In my case, it was throwing errors because of the Windows file paths that we had to migrate to new homes on the Linux server.  Instead of D:\folder\folder\..... we had to change it to /var/www/html/.......

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 14, 2018 Nov 14, 2018

Copy link to clipboard

Copied

All the logs in apache refer to a segmentation fault which is a memory access issue. Our lead programmer has created very small application and has started testing that... only to receive the same errors. I will be reviewing the script tomorrow along with the cf configurations just to see if there is anything that jumps out at me. And then I will attempt to test the connector (which I have no true experience doing). I am curious though as to how many people actually have CF running on a linux (ubuntu for us) based server.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

Plenty of people have CF running on Linux. I don't know how many people have CF 2018 running on Linux yet. This really sounds like a web server connector issue to me.

Dave Watts, Fig Leaf Software

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

We have older CF builds running on a couple different versions of Linux but have yet to get CF2018 stable on Ubuntu. I have reviewed the connector, ran some more updates for Ubuntu and also updated CF2018. We are testing it as we speak. It seems like the apache2 memory allocation is locking up with cf allocated memory. I am not certain as to how to approach this issue. I do think you are correct about the connector and we are continuing to evaluate the situation.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

Well, the problem with Linux is that there's no single Linux environment. Different versions of Apache, different Linux distros, different configurations like SELinux, etc. So that's a thing. Most of the stuff I do is on CentOS, and I haven't done anything with CF 2018 on Linux yet myself.

Dave Watts, Fig Leaf Software

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

Dave, any thoughts as to how to go about troubleshooting this:

[Thu Nov 15 09:59:28 2018] [8553:139938270734080] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

Well, that's definitely a connector issue, in that the connector is identified as "mod_jk" or something along those lines. You could try lowering or raising the number of CF threads. You could try setting a threshold on the number of Apache threads - as I understand it, Apache doesn't have a limit by default. You could make sure you have the latest upgrade for CF 2018, which comes with a new version of Tomcat and therefore a new version of the connector. If you've already done that, you could downgrade to the previous upgrade or the initial install, which will have the old version of the connector.

There appears to be some connection between the thread pooling in mod_jk and the ThreadsPerChild setting in httpd.conf (or whatever other conf file is included - modern Apache has so many now!) Try setting them the same - there's a bunch of files created in the connector directory for each connector instance like connector.properties etc - I'm not in front of a CF/Apache server right now so I don't have exact paths.

If you don't actually need Apache functionality (mod_rewrite, virtual hosts and virtual directories, that sort of stuff), you could just stop Apache HTTPD and use Tomcat as the listener on TCP/80 or TCP/443 - the latter is kind of a pain, though. A lot of people use external web servers when they don't really need them, myself included. That's not really troubleshooting, though.

Good luck! I mean that in all seriousness.

Dave Watts, Fig Leaf Software

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 15, 2018 Nov 15, 2018

Copy link to clipboard

Copied

Dave, I am currently stepping through those configurations... mod_jk and ThreadsPerChild setting as you wrote this... so I guess that means we are on the same page. Hopefully I will find the sweet spot. I'll keep this thread posted.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Nov 16, 2018 Nov 16, 2018

Copy link to clipboard

Copied

I have made a couple changes:

Apache2.... replaced mpm_prefork with mpm_event:

<IfModule mpm_event_module>

        StartServers            2

        MinSpareThreads          25

        MaxSpareThreads          75

        ThreadLimit              64

        ThreadsPerChild          500

        MaxRequestWorkers         150

        MaxConnectionsPerChild   0

</IfModule>

Coldfusion.... worker properties adjustment, added:

    worker.cfusion.max_reuse_connections=250

    worker.cfusion.connection_pool_size=500

II made certain that the connection pool size is the same as the ThreadsPerChild value. We are currently testing the performance looking to see if it can sustain the load. I will most likely revamp the actual configuration values, but using the mpm_event over the mpm_prefork was the start of the solution.

I will follow up as other tweaks are made.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 19, 2018 Nov 19, 2018

Copy link to clipboard

Copied

ricks29770624  wrote

I have a script that, when called from a web browser, returns an empty response.  consider the following code:

-------------------------------------------------------------------

hello

    <cfstoredproc procedure = "ParseAPStoriesExecDTSCentOS" datasource = "#app_datasourceAdminUser#" >

            <cfprocparam type ="OUT" cfsqltype="CF_SQL_INTEGER" variable=OkToContinue dbvarname="@OkToContinue">

                <cfprocresult name="result">

    </cfstoredproc>

    <cfoutput>#result.OkToContinue#</cfoutput>

goodbye

<cfexit>

OK. For the sake of curiosity, what happens when you place the following line before 'hello'?

<cfflush interval="5">

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Nov 19, 2018 Nov 19, 2018

Copy link to clipboard

Copied

I haven't gone back and looked at this since the script was fixed.  What was it that you suspect was happening?  What is cfflush supposed to do here?

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Nov 19, 2018 Nov 19, 2018

Copy link to clipboard

Copied

I suspect a thread is hanging. Cfflush is supposed to flush some letters to the client.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Jan 21, 2019 Jan 21, 2019

Copy link to clipboard

Copied

Did you manage to solve this issue? We are using ColdFusion 2018 Standard (Update 1) together with SLES12 SP3 and face quiet the same problem...

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Mar 04, 2019 Mar 04, 2019

Copy link to clipboard

Copied

We have encountered this issue with CF18 while using Apache 2.4.x prefork on RedHat 7. I'm sharing my experience here to document it and in hopes it may help others.

To summarize:

  • mod_jk is crashing, specifically function: jk_connector_CF_func
  • it looks like I was able to fix this (still testing to confirm) by using MPM Event instead of MPM Prefork

Details:

While setting up severs to run CF18, over the course of a couple months a few times I noticed an ERR_EMPTY_RESPONSE type error message but didn't think much of it. As we got closer to bringing this system live - I started to test longer running URLs and the issue became quite clear.

 

From my testing, I found that pages that took >20 seconds to complete processing had a high likelihood (>50%) that the connection would be dropped - thus producing an ERR_EMPTY_RESPONSE. Page processing greater than 30 seconds had a likelihood close to 100%.

 

Using a long running page to test this issues showed dropped connections anywhere from a few seconds out to 20 or more - but again - by 30 seconds elapsed time there was almost always guarantee of dropped connection and thus generation of ERR_EMPTY_REPONSE. 

 

Adding ouptut and flush statements to the code so that the server would send responses back to the browser on a regular (<=5 seconds) interval had no effect on the dropped connections - however the error changed to ERR_INCOMPLETE_CHUNKED_ENCODING.

 

Ajax calls receiving ERR_EMPTY_RESPONSE would repeat the call one additional time - producing double processing log entries. Once the error changed to ERR_INCOMPLETE_CHUNKED_ENCODING Chrome AJAX calls would issue an error while Firefox AJAX calls stated status was OK (using jquery library). Regardless - the underlying connection issue was unsolved and still an issue. 

 

To eliminate any further double calling - started using curl to test requests going forward.

 

Initial review of Apache log files showed nothing - literally - as anytime a connection was dropped - there would be no corresponding entry in Apache for that request (even though it was processed by Coldfusion). We checked separate backend logs and verified that the requests were coming through to Coldfusion and completing - but for some reason the connection from Apache to the browser was dropping.

 

Eventually I found, as others have noted above, a warning in mod_jk logs that seemed strange:

 

[Thu Feb 28 20:50:00 2019] [9137:139898250012416] [warn] ajp_get_endpoint::jk_ajp_common.c (3705): Unable to get the free endpoint for worker cfusion from 1 slots

 

The timing of the error was correlated to URL requests where we experienced a dropped connection.

 

Turning up debug/trace type logging in Apache led me to this:

 

[Thu Feb 28 19:14:23.317994 2019] [core:notice] [pid 7212] AH00052: child pid 7371 exit signal Segmentation fault (11)

 

Next fire up the GDB debbuger in linux, limit Apache to one process, attach to the Apache process that receives requests, running a CURL test and once it drops connection I see the following output:

 

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7f9778ff9700 (LWP 7439)]
0x00007f9785f0aeb4 in jk_connector_CF_func (thd=<optimized out>, data=<optimized out>) at mod_jk.c:4143
4143 mod_jk.c: No such file or directory.

 

So at this point looks like mod_jk is crashing and because of some issue with the jk_connector_CF_func which clearly looks like it was added by Adobe for integration with Coldfusion (looking at the publicly available source code at line 4143 brings you to the end of the file). 

 

I am still testing this - but it looks like I was able to fix this issue by using MPM Event in Apache - as others here have noted. I am using the following configuration which does not generate any warnings by Apache when checking via configtest:

 

<IfModule mpm_event_module>
        StartServers            2
        MinSpareThreads          25
        MaxSpareThreads          75
        ThreadLimit              500
        ThreadsPerChild          500
        MaxRequestWorkers         500
        MaxConnectionsPerChild   0
</IfModule>

 

I will still be conducting more testing this week - and will update if there are any other significant changes or findings. 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Mar 05, 2019 Mar 05, 2019

Copy link to clipboard

Copied

confirming that through my testing AFAICT the recent CF update (3) from around March 4 2019 does NOT fix this issue.

After applying ColdFusion (2018 release) Update 3 , Coldfusion version now shows as: 2018,0,03,314033

Post applying update 3 testing on servers running Apache 2.4.x...

  1. configured to use MPM prefork continue to fail with either ERR_EMPTY_RESPONSE or ERR_INCOMPLETE_CHUNKED_ENCODING
  2. configured to use MPM event continue to succeed in my tests so far

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Community Professional ,
Apr 09, 2019 Apr 09, 2019

Copy link to clipboard

Copied

Rob, about your saying that the problem still happens after updating CF 2018 to update 3, I'm curious about two things:

  1. Do you actually have the PMT running against CF? And if so, is it on the same machine as CF or another?
  2. And after doing CF2018 update 3, did you also update the PMT (assuming you are running it)? That is not done AUTOMATICALLY by CF. Instead, it's a jar file that must be run manually. I discussed it in more detail in a blog post here: 
    Running the CF 2018 PMT? Have you manually applied the recent update to it?

Indeed, to others reporting this "blank page" problem, who either DO or DO NOT feel that changing the heartbeat helped, can you clarify if you DO have the PMT running against the CF instance in question?


/Charlie (server troubleshooter, carehart.org)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines