Skip to main content
N
nebbb
Participating Frequently
August 30, 2012
Question

FMS 4.5.x: The FLV segment cache is full

  • August 30, 2012
  • 4 replies
  • 5369 views

We just migrated our service to FMS 4.5.  Since our new server is in production, we started having the following error message in the core.log files:

The FLV segment cache is full

There is very little information about this problem on the web.  The only information found is to increase the value of parameter SERVER.FLVCACHE_MAXSIZE in the fms.ini file.  We tried this:

SERVER.FLVCACHE_MAXSIZE=2000

Increased the cache size only delayed the problem from a few hours to 1 day.  It seems that FMS 4.5 is filling up the cache without cleaning anything.  Note that we are doing a very high number of video recordings.  The only way to solve the issue is to restart the FMS service.

One week ago, we were running FMS 3.5.7 on Windows 2003 and we never had this issue.  FMS was running with 500Mb of cache and was working fine for years.  This bugs has appeared since we moved to the FMS 4.5 on a Windows 2008 server.

Coincidentally, Adobe released the 4.5.3 patch that includes fix for memory problems one week ago.  But even this version has the issue.

I saw a similar thread here: http://forums.adobe.com/thread/913107

Anyone can point to a solution?  This is a major bug.

    This topic has been closed for replies.

    4 replies

    stoem
    stoem
    Known Participant
    November 10, 2014

    My friend Jorge asked me to post this reply as a potential solution.

    This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

    1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

    2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

    CONFIG FILE Server.xml

    line 251:    <AutoCloseIdleClients enable="true">

    line 264:    <FLVCachePurge>10</FLVCachePurge>

    line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

    line 317:    <ApplicationGC>1</ApplicationGC>

    line 319:    <FLVCacheSize>25</FLVCacheSize>

    line 594:    <CoreGC>30</CoreGC>

    CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

    line 74:        SERVER.FLVCACHE_MAXSIZE=3000

    NOTE: This value (in MB) should be no more than 25% of total RAM.

    3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

    sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

    Bottom line: allow a core to go idle so that it unloads and releases memory.

    Hope that helps.

    H
    Adobe Employee
    hparmar
    Adobe Employee
    November 10, 2014

    Also you can configure Rollover for cores in Application.xml...See

        <LifeTime>

          <RollOver></RollOver>

          <MaxCores></MaxCores>

        </LifeTime>

    in Application.xml

    Also http://www.adobe.com/devnet/adobe-media-server/articles/hardening_guide.html

    stoem
    stoem
    Known Participant
    November 10, 2014

    My friend Jorge asked me to post this reply as a potential solution.

    This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

    1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

    2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

    CONFIG FILE Server.xml

    line 251:    <AutoCloseIdleClients enable="true">

    line 264:    <FLVCachePurge>10</FLVCachePurge>

    line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

    line 317:    <ApplicationGC>1</ApplicationGC>

    line 319:    <FLVCacheSize>25</FLVCacheSize>

    line 594:    <CoreGC>30</CoreGC>

    CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

    line 74:        SERVER.FLVCACHE_MAXSIZE=3000

    NOTE: This value (in MB) should be no more than 25% of total RAM.

    3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

    sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

    Bottom line: allow a core to go idle so that it unloads and releases memory.

    Hope that helps.

    J
    jim_houseman
    Known Participant
    October 27, 2014

    It's now coming on the end of October. Any answer? @ShivenK?

    S
    ShivenK
    Inspiring
    September 5, 2012

    Hi,

    Please go through this piece of information and configure the parameters accordingly:

    http://help.adobe.com/en_US/flashmediaserver/configadmin/WS5b3ccc516d4fbf351e63e3d119f2925e64-7ff0.html#WS5b3ccc516d4fbf351e63e3d119f2925e64-7fee

    Also, let me know:

         1.     How many streams are being published onto the server?

         2.     What is the container and codec for the published streams?

    Thanks,

    Shiven

    N
    nebbbAuthor
    Participating Frequently
    September 5, 2012

    Hello Shiven,

    thank you for the reply.  A few days back, we increased SERVER.FLVCACHE_MAXSIZE to 2500, and also found that we can clean up the memory faster by setting FLVCachePurge to 1.  We also figured that the old FMS 3.5.7 probably would have had the same issue.  In fact, we saw that the same week we changed to our new server, we also had an increase in usage.

    However, this still doesn't explain that the memory fills up so quickly.  It seems that recording videos is a very heavy memory thing for FMS.  Here are the answers to your questions:

    1) we had about 35,000 streams on a day that the cache filled up

    2) we use standard H.263 in FLV container with NellyMoser mono.

    We actually were able to reproduce the problem on any FMS version by setting SERVER.FLVCACHE_MAXSIZE to 1 and doing a few recordings.  After about 5 recordings (of 30 seconds each), the cache problem appears.  By setting the FLVCachePurge to 1, it's harder to reproduce as the cache is cleaned faster.  However, I would expect a cache system to clean up the oldest stream recorded no matter if the caching period is over or not.  There is certainly a major issue to look at here for your team, as we can't expect to just increase the cache size each time the problem re-appear.

    Currently, in addition to the two changes that we did in the configuration, we also do a reboot of the FMS service every night.  This is a temporary measure and we need a real solution to this issue.  We can't be rebooting FMS as a final solution.

    Ask your team to test and reproduce the cache problem following these steps:

    a) set SERVER.FLVCACHE_MAXSIZE=1 in fms.ini

    b) record a few stream to the FMS, each of about 30 seconds

    c) notice that the cache generate the error and that it's impossible to record another stream until the FMS service is rebooted

    EXPECTED behavior: clear the oldest stream, no matter the time set in FLVCachePurge

    Please follow-up on this thread with more explaination.

    Thank you.

      A
      Anonymous
      October 18, 2012

      Hi Nebbb,

      Did you manage to get this problem under control? We're seeing exactly the same problem after migrating to 4.5, and tweaking those settings has not made it go away.

      Best regards,

      Jorge

      Terms & ConditionsAccessibility statement
      Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices