Skip to main content
J
jarviswabi
Inspiring
July 18, 2019
Question

Performance Monitoring Toolset keeps crashing

  • July 18, 2019
  • 2 replies
  • 2059 views

I've been having lots of problems with the Performance Monitoring Toolset since putting my CF2018 Enterprise intranet into production about a month ago.  I have the PMT installed on my Test/Dev server (Windows Server 2019, 64Gb RAM, running a Development instance of CF18 plus the PMT), and it's monitoring my Dev site plus my production site on another server (same hardware/setup).

What seems to keep happening every couple days or so is that the Elasticsearch datastore services that underlie the PMT are crashing, which initially makes it impossible to login.  When I find that, I go to restart the Datastore service, then the PMT service itself, but the PMT service won't start because it fails to connect to the Datastore.  After several attempts, I usually get frustrated and uninstall and reinstall the whole thing, which is clearly not a viable long-term solution.

The datastore/logs files show stuff like:

org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed

java.lang.OutOfMemoryError: Java heap space

I know the latter, so on my most recent install, I bumped the heap size up to 4Gb, thinking maybe it's just not enough to keep track of my workload (seems unlikely).

The PMT logs show this when it won't start:

[ERROR] 2019-06-27 11:45:52.658 com.adobe.pms.es.client.ElasticSearchClient - Datastore Service not available. Retrying to connect...

[ERROR] 2019-06-27 11:46:27.884 com.adobe.pms.es.client.ElasticSearchClient - Datastore Service not available. Shutting down Performance Management Suite...

Even though the Datastore service seems to be running and its own logs show it having recovered.

I was hoping there'd be more mentions of these issues online or a patch, but I can't find anything related to it.  It's concerning because I keep finding myself needing to be able to see what's running on the Production server and the PMT is down and so I have no visibility to my server's active jobs.  Seems like the PMT was put together a bit hastily on a new stack of technologies that don't exactly work well together, and not having the Server Monitor any longer leaves us a bit blind as to what our servers are doing.

If anybody's had similar experience with the PMT or any insight on how to manage it, I'd sure appreciate it.

    This topic has been closed for replies.

    2 replies

    BKBK
    Community Expert
    BKBKCommunity Expert
    Community Expert
    August 4, 2019

    I looked into this further. The question of the JVM settings for Elasticsearch is more complex than I at first thought.

    First, let's do away with a logical fallacy. My PMT installation was already working properly. So if it continued working after I changed the Xms and Xmx settings in jvm.options, that wouldn't mean that such a change had an effect. In fact, when I examined the file C:\ColdFusion2018PerformanceMonitoringToolset\datastore\logs\elasticsearch-service-x64-stdout.2019-08-04.log afterwards, I noticed that the settings in use were still Xmx=2g and Xms=2g.

    Apparently, neither jvm.options nor elasticsearch-service.bat should be used to set the values of Xmx and Xms. The purpose of the file elasticsearch-service.bat is to create the start-up environment required during the Elasticsearch installation. The contents of the file elasticsearch-service.bat actually suggest that

    1. If JAVA_HOME is defined as an environment variable, the Elasticsearch installation will use that JVM;
    2. You may define the Xmx and Xms values as the environment variable ES_HEAP_SIZE.
    3. The start-up process writes the file jvm.options to disk.
    B
    beekermd03
    Participating Frequently
    July 29, 2020

    All,

     

    I had been battling this one for a bit.  I was quite confused but figured it out.

     

    System details

    • Running CF18 on Windows Server 2016.
    • Running the latest version of PMT as Charlie recommended

     

    Problem

    I was expericing non-stop "com.adobe.pms.es.eshealth.ESStatCalculator - Heap utilization for Elasticsearch node PmV4wph is 91%" and then "com.adobe.pms.es.eshealth.ESStatCalculator - Elasticsearch node PmV4wph recovered from high Heap utilization" alerts.

     

    What I tried to fix it

    1. I tried to increase the MAX JVM heap as indicated in the "ColdFusion2018PerformanceMonitoringToolset\datastore\config\jvm.options" file to -Xms4 and -Xmx4.
    2. Toggled the ColdFusion 2018 Performance Monitoring Toolset Datastore Service multiple times and monitored the RAM consumption using Task Manager for the "Commons Daemon Service Runner" but it never went beyond ~2GB.
    3. I tried as far as deleting the jvm.options file.  Didn't matter.  The service doesn't use it.

     

    What fixed it

    FINALLY found a stackoverflow blog:  https://stackoverflow.com/questions/28798845/how-to-set-memory-limit-to-elasticsearch-in-windows

    1. From a command prompt.  Navigated to the appropriate datastore\bin directory and ran the following: elasticsearch-service.bat manager
    2. Went to the Java tab
    3. Updated the Initial and Max memory pool sizes input boxes.
    4. Applied.
    5. Restarted the service.
    6. Check back in the Task Manager and the Commons Daemon Service Runner is now at 4 GB!! Voila!

     

    I hope this helps someone out there!  I struggled for a bit with it!!

      B
      beekermd03
      Participating Frequently
      July 29, 2020

      Oh and it looks like the version of Elastic Search PMT is running is Kibana 5.6.3 for those that are curious.  At the time of this writing, Elastic Search is at 7.8.1.

       

      That being said, no way CF can package up a 2020 Elastic Search version in the CF 2018 release which I totally get.  Just a point to note.  Not holding anything agaist CF. 

       

      Kibana was released on October 10, 2017 so that was most likely a current version when they packaged up PMT.

        BKBK
        Community Expert
        BKBKCommunity Expert
        Community Expert
        July 28, 2019

        From what I've seen online, the combination of Out-Of-Memory and  "org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed" suggests that your installation has too many shards. Get the Elasticsearch properties, for example, using cfhttp-get. See, for example,

        https://discuss.elastic.co/t/elasticserach6-1-1-restart-and-i-got-all-shards-failed/117155

        According to one of the links,

        How many shards should I have in my Elasticsearch cluster? | Elastic Blog

        "A good rule-of-thumb is to ensure you keep the number of shards per node below 20 per GB heap it has configured. A node with a 30GB heap should therefore have a maximum of 600 shards"

        J
        jarviswabiAuthor
        Inspiring
        July 29, 2019

        Thanks, that's helpful.  I think I stumbled into this solution (at least temporarily) when I managed to increase the JVM heap size from 2Gb to 4Gb (found I had to use ColdFusion2018PerformanceMonitoringToolset\datastore\bin\elasticsearch-service.bat manager GUI to change that value), which I guess gives me room for more shards.  PMT has stayed running now for 6 days.  I am seeing older indexes, so I'll see how archiving works, but for now, it seems stable.  Hopefully soon there will be more information on how to better manage this tool, since it's pretty critical to enterprise operations running CF2018.

        BKBK
        Community Expert
        BKBKCommunity Expert
        Community Expert
        July 30, 2019

        jarviswabi  wrote

        ...I managed to increase the JVM heap size from 2Gb to 4Gb (found I had to use ColdFusion2018PerformanceMonitoringToolset\datastore\bin\elasticsearch-service.bat)

        I agree with an increase in heap size. But do you mean perhaps C:\ColdFusion2018PerformanceMonitoringToolset\datastore\config\jvm.options ?

        I think that that is the best place to modify the heap size. Just change the settings from

        -Xms2g

        -Xmx2g

        to

        -Xms4g

        -Xmx4g

        Terms & ConditionsAccessibility statement
        Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices