Skip to main content
Bardnet
Inspiring
Dec 10, 2020 10:24 PM UTC
Question

"Phantom" calls to long running scripts (connector issue)

Hi,

 

I find myself in a ColdFusion 2018 environment on a Linux server behind an Apache server.

I had issues and tried to track them down. It turned out my long running script is called over and over again.

I call it manually through a webbrowser once. I see it initiates only one request through the browser's F12 tool.

After I realized that requests multiply I searched the internet for possible solutions and found this thread on this site: https://coldfusion.adobe.com/2019/02/apache-crashes-serving-multiple-thread-requests-using-cf2018/

 

I eventually set heartbeat_interval to 0 and noticed no change. Only after httpd was restarted the "Segmentation fault" messages stopped.

 

Now I start my long running script and enter some data into a database when it is started, including a timestamp. This is how I see that the script is started again and again every 6 minutes.

I guess this is still a connector issue.

Does this behaviour ring a bell? I like to believe some instance (connector, httpd) sees no response from CF's Tomcat and repeats the request. Is there such a timeout? The Request Timeout in CFAdmin is disabled - the checkbox is unchecked.

 

Best,

Bernhard

    This topic is closed to new replies. Start a new post to keep the conversation going.

    9 replies

    paule12345
    Inspiring
    Dec 18, 2020 6:55 PM UTC

    We've observed the same behavior on our CF2016 Enterprise install on Windows 2012.   We see CF scripts that take longer than 6 minutes spawn a duplicate thread.   I can see it in FusionReactor, at approx 377,000 ms, a duplicate thread will fire up.   cfsetting requesttimeout has no effect.   Timeout settings in CF admin have no effect.   IIS app pool recycling is not causing it.  This is not a scheduled task.

    Translate
    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    Dec 19, 2020 3:25 PM UTC

    That this has been observed on CF2018+Linux+Apache and CF2016+Windows+IIS suggests an issue in the ColdFusion engine. So I wonder whether we can reproduce it. For example, using the following code:

     

    <cfscript>
        // This request will run for 9 minutes
	setting requestTimeout=540;
	sleep(540000);
</cfscript>

     

    Steps:

    1) Run the above code in a CFM page.

    2) Wait 6 minutes. Then use the jstack tool of the Java that ColdFusion is running on to take ColdFusion's thread-dump.

    For example, my ColdFusion installation runs on C:\Program Files\Java\jdk-11.0.8, as Windows process 22816. So, I navigate in DOS to C:\Program Files\Java\jdk-11.0.8\bin, and take thread-dumps using the following DOS command (the dump is saved as a text file on my desktop):

     

    jstack -l 22816 > C:\Users\BKBK\Desktop\CF_threadDump_1.txt

     

    3) Continue taking thread-dumps every 20 to 30 seconds, renaming subsequent files CF_threadDump_2.txt,  CF_threadDump_3.txt, ... till the end of the request.

    4) Study the dumps to identify any additional or extraneous threads. 

     

     

    Translate
    paule12345
    Inspiring
    Dec 19, 2020 5:39 PM UTC

    BKBK, I tried your code today and the script ran to completion.  It did not spawn a duplicate thread after 6 minutes.   Something else in our code must be causing it.  I'll have to do some more testing with our actual production code.

    Translate
    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    Dec 13, 2020 8:32 PM UTC

    Any error messages in coldfusion-error.log and coldfusion-out.log?

    Translate
    BardnetAuthor
    Inspiring
    Dec 17, 2020 11:21 PM UTC

    Sadly, the log files are not helpful.

    Translate
    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    Dec 11, 2020 1:40 PM UTC

    Are you running any scheduled tasks?

    Translate
    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    Dec 11, 2020 1:35 PM UTC

    How long does your long-running script run for? Let's say 15 minutes.

     

    Then you could try the following. In the Administrator, go to Server Settings > Settings and set Timeout Requests after seconds to 1200.
    Then proceed to Server Settings > Request Tuning and set Queue Timeout Settings to 1200.

    Restart ColdFusion, just to be sure the change kicks in.

    Any joy?

    Translate
      BardnetAuthor
      Inspiring
      Dec 17, 2020 11:21 PM UTC

      Thanks alot for your pointers! I  try this setting.

      As a quick fix I tried to identify if the script is already running and cancel every other request.

      Still looking for the issue, bud I could limit the negative effects.

      Translate
      Terms & ConditionsAccessibility statement
      Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices