"Phantom" calls to long running scripts (connector issue)

[Bardnet]

Hi,

I find myself in a ColdFusion 2018 environment on a Linux server behind an Apache server.

I had issues and tried to track them down. It turned out my long running script is called over and over again.

I call it manually through a webbrowser once. I see it initiates only one request through the browser's F12 tool.

After I realized that requests multiply I searched the internet for possible solutions and found this thread on this site: https://coldfusion.adobe.com/2019/02/apache-crashes-serving-multiple-thread-requests-using-cf2018/

I eventually set heartbeat_interval to 0 and noticed no change. Only after httpd was restarted the "Segmentation fault" messages stopped.

Now I start my long running script and enter some data into a database when it is started, including a timestamp. This is how I see that the script is started again and again every 6 minutes.

I guess this is still a connector issue.

Does this behaviour ring a bell? I like to believe some instance (connector, httpd) sees no response from CF's Tomcat and repeats the request. Is there such a timeout? The Request Timeout in CFAdmin is disabled - the checkbox is unchecked.

Best,

Bernhard

[BKBK]

How long does your long-running script run for? Let's say 15 minutes.

Then you could try the following. In the Administrator, go to Server Settings > Settings and set Timeout Requests after seconds to 1200.
Then proceed to Server Settings > Request Tuning and set Queue Timeout Settings to 1200.

Restart ColdFusion, just to be sure the change kicks in.

Any joy?

[Bardnet]

Thanks alot for your pointers! I  try this setting.

As a quick fix I tried to identify if the script is already running and cancel every other request.

Still looking for the issue, bud I could limit the negative effects.

[BKBK]

Are you running any scheduled tasks?

[BKBK]

Any error messages in coldfusion-error.log and coldfusion-out.log?

[Bardnet]

Sadly, the log files are not helpful.

[paule12345]

We've observed the same behavior on our CF2016 Enterprise install on Windows 2012.   We see CF scripts that take longer than 6 minutes spawn a duplicate thread.   I can see it in FusionReactor, at approx 377,000 ms, a duplicate thread will fire up.   cfsetting requesttimeout has no effect.   Timeout settings in CF admin have no effect.   IIS app pool recycling is not causing it.  This is not a scheduled task.

[BKBK]

That this has been observed on CF2018+Linux+Apache and CF2016+Windows+IIS suggests an issue in the ColdFusion engine. So I wonder whether we can reproduce it. For example, using the following code:

<cfscript>
        // This request will run for 9 minutes
	setting requestTimeout=540;
	sleep(540000);
</cfscript>

Steps:

1) Run the above code in a CFM page.

2) Wait 6 minutes. Then use the jstack tool of the Java that ColdFusion is running on to take ColdFusion's thread-dump.

For example, my ColdFusion installation runs on C:\Program Files\Java\jdk-11.0.8, as Windows process 22816. So, I navigate in DOS to C:\Program Files\Java\jdk-11.0.8\bin, and take thread-dumps using the following DOS command (the dump is saved as a text file on my desktop):

jstack -l 22816 > C:\Users\BKBK\Desktop\CF_threadDump_1.txt

3) Continue taking thread-dumps every 20 to 30 seconds, renaming subsequent files CF_threadDump_2.txt,  CF_threadDump_3.txt, ... till the end of the request.

4) Study the dumps to identify any additional or extraneous threads. 

[paule12345]

BKBK, I tried your code today and the script ran to completion.  It did not spawn a duplicate thread after 6 minutes.   Something else in our code must be causing it.  I'll have to do some more testing with our actual production code.

[BKBK]

Hi paule12345

Thanks for the update. Could you please share the stacktrace of the extra thread (the stracktrace you get from Fusionreactor).