Skip to main content
-
-JaBo-
Participant
September 28, 2020
Answered

ColdFusion 2018 - TIMED_WAITING threads stacking up endlessly?

  • September 28, 2020
  • 2 replies
  • 3072 views

Hi everyone,

 

before I start here is some basic info about our production environment:

OS: CentOS Linux 7

CF: ColdFusion 2018 (2018.0.10.320417)

Java: 11.0.8

FusionReactor: 8.5.0

Apache httpd: 2.4.6

mod_jk: 1.2.46

 

We have installed ColdFusion 2018 lately and noticed that FusionReactor shows a lot of TIMED_WAITING threads called `pool-<number>-thread-1` which ramp up and don't seem to be closed:

since this looks worrying to me I'm here to ask if anyone knows if this is normal behaviour or if we need to take action before our server crashs like this one.

 

I took a look at some thread dumps of these `pool-*-thread-1`-threads but they don't give me a clue where they come frome and what they were supposed to be used for.

 

Example:

 

I don't know much about Java threads yet and thus can not debug this right away. Maybe anyone here has an advice where to start?

 

btw: at first I thought these threads are the ones which are configured in `workers.properties` and `server.xml` but they are set to max 250 whereas those pool-threads are thousands - anyways here is our workers.properties for reference:

 

worker.cfusion.connection_pool_timeout=60
worker.cfusion.connection_pool_size=250
worker.cfusion.max_reuse_connections=250

and the connector tag 

<Connector connectionTimeout="60000" maxThreads="250" ...

 

Cheers,

Sebastian

    This topic has been closed for replies.
    Correct answer Charlie Arehart

    Jabo, this is indeed a problem that has hit others, like that thread you linked to shows, and as I have also experienced with folks I've helped directly (in my consulting).

     

    And I just offered a comment on that other thread that I will offer now here (since some may not bother going there, and you yourself may not have had any reason to know I added the comment there since writing the above):

     

    "...just last week I heard of a bug fix that seems may POSSIBLY be related to this, at least to a blossoming of threads in CF2018. Unfortunately, the bug report is not public, but one CAN request the bug fix from Adobe. I have asked them to clarify if it's about "pool" threads specifically, but have not heard back since last week.

     

    If you or others want to either give the fix a try, or ask Adobe for more info, the hotfix file name you'd request is hf201800-4207395.jar. (If they would want to share the file or a link to it here, I will leave that to them.)

     

    With hofixes like this, you would drop it into the lib/updates folder under cfusion (or whatever folder name represents any other CF instance you are running), and do NOT remove any other already there, then restart CF. It should pull that new fix in, adding it to any other update already there. And then, if a later update would incorporate this fix, that update will itself remove this hf jar from the folder for you.

     

    If anyone tries it, I hope you will report back on how things go. I have shared the same news with the clients who experiecned it, and I hope to remember to report back if I hear news from them.

     

    Or if you or anyone else may have since tried and found resolution some other way, please do let us know

    "

     

    Same goes for you, Jabo, or anyone else seeing this here. I would LOVE to see this resolved for everyone concerned.

    2 replies

    T
    tribule
    Legend
    March 4, 2021

    We have a similar issue to Dirk, but we do not use CFLOGIN. Rising pool threads over time, slowly. We have had to increase the number of threads in our connector to a very large number to allow the server to stay online, which is far from ideal.

    We applied the Adobe hotfix to see if it helped, but alas it did not.  On this blog - https://blog.tier1app.com/2014/09/30/memory-leak-in-java-executor/ - there is a statement that the Java/TomCat Executor could be buggy, and that "parked" threads are never properly released. This article is from 2014.

     

    Can someone tell me (Adobe?) if that issue has been resolved? Was it a bug? The article says to use the shutdown() method in the Executor, as this will release the parked threads. Does anyone know how we can call that method on our Executor so we can try? We use the default Executor - the internal thread pool (https://docs.oracle.com/javase/10/docs/api/java/util/concurrent/ThreadPoolExecutor.html). We have no <Executor> entry in server.xml configured.

    Advice appreciated as always.

    Charlie Arehart
    Community Expert
    Charlie ArehartCommunity Expert
    Community Expert
    March 5, 2021

    That 2014 post doesn't identify a tomcat bug. It identifies a Java coding example that lead to a blossoming of pool- threads.

     

    As for any hope that we as cf users can fix this based on the Java "shutdown" code offered, I find that highly unlikely.

     

    These cases of pool thread leaks seem entirely caused by something inside of cf (or a library it leverages). That could be tomcat, sure, but the cf login issue discussed earlier had nothing to with tomcat.

     

    Your issue could be anything, really. I doubt it's caused by any code on your end, but some code (that only Adobe can help with, as a bug fix, I'd expect ) or perhaps some config in cf that we MAY be able to change  (once we know what the problem is). 

     

    The challenge of course is determining the cause. 

    /Charlie (troubleshooter, carehart. org)
    Charlie Arehart
    Community Expert
    Charlie ArehartCommunity ExpertCorrect answer
    Community Expert
    September 28, 2020

    Jabo, this is indeed a problem that has hit others, like that thread you linked to shows, and as I have also experienced with folks I've helped directly (in my consulting).

     

    And I just offered a comment on that other thread that I will offer now here (since some may not bother going there, and you yourself may not have had any reason to know I added the comment there since writing the above):

     

    "...just last week I heard of a bug fix that seems may POSSIBLY be related to this, at least to a blossoming of threads in CF2018. Unfortunately, the bug report is not public, but one CAN request the bug fix from Adobe. I have asked them to clarify if it's about "pool" threads specifically, but have not heard back since last week.

     

    If you or others want to either give the fix a try, or ask Adobe for more info, the hotfix file name you'd request is hf201800-4207395.jar. (If they would want to share the file or a link to it here, I will leave that to them.)

     

    With hofixes like this, you would drop it into the lib/updates folder under cfusion (or whatever folder name represents any other CF instance you are running), and do NOT remove any other already there, then restart CF. It should pull that new fix in, adding it to any other update already there. And then, if a later update would incorporate this fix, that update will itself remove this hf jar from the folder for you.

     

    If anyone tries it, I hope you will report back on how things go. I have shared the same news with the clients who experiecned it, and I hope to remember to report back if I hear news from them.

     

    Or if you or anyone else may have since tried and found resolution some other way, please do let us know

    "

     

    Same goes for you, Jabo, or anyone else seeing this here. I would LOVE to see this resolved for everyone concerned.

    /Charlie (troubleshooter, carehart. org)
      dirk.
      dirk.
      Participating Frequently
      September 29, 2020

      Hi Charlie, first of all thank you so much for your offer to help us investigating this issue. It is so frustrating to restart Coldfusion every two to three hours, when the system becomes completely unresponsive. We only have one instance running but thank you for your hint to double-check the installation of the hotfix. To be honest, I didn't compare the filenames, it seems they obviously sent me the wrong patch (hf201800-4207069.jar, not hf201800-4207395.jar). We are still waiting to get the correct file. Provided that the fix comes soon, I will post the results tomorrow morning.
      Thanks, Dirk

      dirk.
      dirk.
      Participating Frequently
      October 1, 2020

      Happy to help, Dirk, and thanks for the kind regards.

       

      About that other hotfix, that's another issue of course ( https://tracker.adobe.com/#/view/CF-4207069 ), and it's been indicated as helping folks with issues of getcomponentmetadata taking a long time, which some are saying was crippling Mura apps on CF2018 (so I am waiting also to hear from folks there about whether it did help that).

       

      But it's definitely not the same problem as this, about threads (especially pool threads) growing at ever increasing rates, fixed by hf201800-4207395.jar. Again, sadly the Adobe bug report for this threads issue was deemed a private one and so we in the public can't see it to know more. But we can hope it will be incorporated into any next update to CF2018, like perhaps update 11.

       

      Am looking forward to hearing what you may find.


      It looks like that the hotfix solved the issue. We installed it a day ago and now everything looks good so far.

       


      I asked the support team about WHAT caused the surge of threads as it obviously does not affect a lot of installations. Unfortunately I just got a very short and useless answer:
      "The issue was that the cf threads were growing exponentially and with this patch we have fixed that."

        Terms & ConditionsAccessibility statement
        Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices