Skip to main content
-Chris
-Chris
Known Participant
December 7, 2023
Question

CF2023 threads growing to limits

  • December 7, 2023
  • 3 replies
  • 5371 views

Having some issues with multiple Ubuntu servers recently upgraded from CF2018 to CF2023, all using jdk-17.0.9, thread counts are growing until they hit their limits with cfsearch errors:

 

java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached at java.base/java.lang.Thread.start0(Native Method) at java.base/java.lang.Thread.start(Thread.java:811) at org.apache.http.impl.client.IdleConnectionEvictor.start(IdleConnectionEvictor.java:96) at org.apache.http.impl.client.HttpClientBuilder.build(HttpClientBuilder.java:1238) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:337) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:348) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:286) at org.apache.solr.client.solrj.impl.HttpClientUtil.createClient(HttpClientUtil.java:273) at org.apache.solr.client.solrj.impl.HttpSolrClient.<init>(HttpSolrClient.java:204) at org.apache.solr.client.solrj.impl.HttpSolrClient$Builder.build(HttpSolrClient.java:968) at coldfusion.tagext.search.SolrUtils._getSolrClient(SolrUtils.java:1740) at coldfusion.tagext.search.SolrUtils.getSolrClient(SolrUtils.java:1688) at coldfusion.tagext.search.SolrUtils.getSolrDocCount(SolrUtils.java:270) at coldfusion.tagext.search.SearchTag.doSolrSearch(SearchTag.java:378) at coldfusion.tagext.search.SearchTag.doSearch(SearchTag.java:252) at coldfusion.tagext.search.SearchTag.doStartTag(SearchTag.java:190) at coldfusion.runtime.CfJspPage._emptyTcfTag(CfJspPage.java:5088) at cfsearch2ecfc1011696301$funcPERFORMSEARCH.runFunction

 

The thread count before the errors start depends on the DefaultTasksMax setting on the server, where it's lower the errors start earlier, some have over 15,000 threads with a higher limit.

 

Any advice on this is much appreciated.

    This topic has been closed for replies.

    3 replies

    A
    AARON31384707wi24
    New Participant
    July 19, 2024

    we are now over 6 months, and Adobe have not realeased a fix?

      -Chris
      -ChrisAuthor
      Known Participant
      September 16, 2024

      The bug has now been closed with Fixed In Build: 2025.0.0.331047. Hopefully not too much longer before it's release.

      BKBK
      BKBK
      Braniac
      September 17, 2024

      Which bug is this? Could you please share the Tracker URL?

      Charlie Arehart
      Charlie Arehart
      Braniac
      March 25, 2024

      Chris (and Ollie), I still think there's value in looking into the solr server rather than cf. Cf's the CLIENT, and while that MAY be where the problem is, it could instead be in the server.

       

      You had said you were using the version implemented by the addons service. (Is that the case for you, Ollie?) And was that indeed the cf2023 addons installer? It doesn't have to be, though it's generally wise.

       

      More important, as for monitoring solr itself, I'd proposed in early Dec here in this thread that you consider implementing fusionreactor ON solr (that addons service) itself, or using jvm tools to monitor it. You replied about having implemented it in cf instead. (And since you have cf and solr on the same machine, note that the licensing of fr is per server, not per app.)

       

      I'm pressing this because we can't know if the problem is indeed on the cf end or the solr end: the hanging evictor threads in cf may be caused by something hanging up or acting differently in the solr side.

       

      Finally, as I offered in that reply, if you (or Ollie or anyone experiencing this issue) might want help diagnosing it, I'm available to help via remote screenshare on a consulting basis.

       

      I realize you've opened a ticket with Adobe, and as I said if they might solve things with you/for you, great. If not, perhaps I can help. You won't pay for time you don't find to be valuable. Recall so that Adobe said in the ticket that they couldn't recreate the problem. This is why I wonder if there may be something we find on your end that they couldn't/didn't replicate. 

      /Charlie (troubleshooter, carehart. org)
      -Chris
      -ChrisAuthor
      Known Participant
      March 26, 2024

      Thanks @Charlie Arehart, solr was installed with the CF2023 add-ons installer. From memory I also tested using a CF2023 installer that had solr included, with the same result.

      I'll look into doing some solr monitoring when time permits, but have been hoping for an Adobe bugfix for this. Although they hadn't replicated it in their first reply to the bug, after I provided some more info the bug status changed to 'to fix' with reason code BugVerified. The status has since changed to 'to test' with reason code NeedMoreInfo. I haven't had any further updates from Adobe, since they asked to archive my ticket and advised they were working on it.

      I'll also just mention how easy it is to replicate, with a fresh install of CF2023 and solr, adding a solr collection from the CF admin and a test page with cfsearch, populating the collection is not required. I've seen the same result in Ubuntu 18, 20 and 22, monitoring the thread growth at a terminal with htop.

      Charlie Arehart
      Charlie Arehart
      Braniac
      March 26, 2024

      Chris, I appreciate how you feel this is easily replicated, with the implication being I try it myself. But both yesterday when I replied and now, I'm seeing this on my phone so cannot. And as much as l might hope to try on a computer, the day got away from me yesterday and may today.

       

      FWIW, bkbk has not confirmed replicating it, but it's not clear if he's tried. And while we can infer from the ticket that Adobe has, since the status has changed it's also not clear. So I'm just saying that while you'd think anyone/everyone should be hitting it, there may well be something more unique about your situation than currently meets the eye. (I do appreciate you could assert that everyone IS hitting it and just doesn't know it.)

       

      Let's hope we learn more, whether from more investigation from Adobe, or any of us, nor you if you opt to explore using FusionReactor to watch things from the solr side (it can give you insights into requests, threads, resources, and much more, and has a 14-day free trial). 

      /Charlie (troubleshooter, carehart. org)
      Charlie Arehart
      Charlie Arehart
      Braniac
      December 7, 2023

      To help those who may aid in diagnosis, please check the values for the "solr server" as indicated 

      in the cf admin: do they (server and port) point to a solr implementation on the same machine as cf? Or another? And do you know if it is the one implemented by the cf2023 installer? Or if by a separately downloaded cf addons-installer, and if so was that the 2023 version? 

       

      You can also check for and provide here the solr version info (and what Java it's using) via a helpful solr admin ui that many never notice: form a url by using that solr server name and portl, such as http:servername:port, and visit that. It should show a link for solr, and clicking that should show a useful solr admin ui.

       

      I do realize that someone recognizing your problem readily may NOT need this info, but while we await that, it's something you can do that MAY prove helpful. 

      /Charlie (troubleshooter, carehart. org)
      -Chris
      -ChrisAuthor
      Known Participant
      December 7, 2023

      Thanks for your reply Charlie.

       

      Solr is running on the same machines as CF, and was installed with the CF2023 add-ons installer. Solr version is 8.11.2, it's Java runtime is 17.0.6+9-LTS-190.

       

      The solr admin is good for showing memory usage but I haven't seen any stats for thread usage unfortunately.

      Charlie Arehart
      Charlie Arehart
      Braniac
      December 7, 2023

      OK on all that. To be clear, I wasn't proposing that the Solr Admin would HELP with the thread usage problem (though there are ways it might). I was merely proposing using it as a means to know a) what does CF point to, b) does that respond as a web site, and c) what did it report for the version info. I asked all that for the sake of others who may help.

       

      Finally, as for finding out what the thread usage (in CF) is about, I would propse that you might find more about that from other tools, whether JVM tools (like jconsole, jvmd, jvisualvm, jmc, etc.) or a tool like FusionReactor (the wonderful CF monitor--which can monitor MORE than just CF, to include that addon/Solr service).

       

      But even then, such tools may not readily tell us WHY CF is using the threads, just by viewing them. But by viewing them over history and connecting the rise in threads to CF request processing, one may be able to connect dots. FR would be especially valuable for that. And while I can help one do that, it's not the sort of work to be done via a forum thread like this. It would be best done as an online screen-share session, which I can offer via my consulting (carehart.org/consulting). 

       

      But I realize you may feel this is a bug in CF, and as such you may prefer to wait for Adobe to respond and help--whether with a ready answer if they already undersand the problem as reported by others, or if they might want to arrange for additional diagnostics from you. Again, I asked what I did above to anticipate that somewhat.

       

      I'll say finally that if you get no reply from them here, consider either sending an email to them (cfinstall@adobe.com) or open a bug report at tracker.adobe.com. Those are sure to get their attention, whereas here they may or may not reply (they often do, but there's no guarantee.)

       

      Hope all that's helpful, to you or others who may find this thread.

      /Charlie (troubleshooter, carehart. org)
      Terms & ConditionsAccessibility statement
      Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices