• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
Locked
1

FMS 4.5.x: The FLV segment cache is full

Explorer ,
Aug 30, 2012 Aug 30, 2012

Copy link to clipboard

Copied

We just migrated our service to FMS 4.5.  Since our new server is in production, we started having the following error message in the core.log files:

The FLV segment cache is full

There is very little information about this problem on the web.  The only information found is to increase the value of parameter SERVER.FLVCACHE_MAXSIZE in the fms.ini file.  We tried this:

SERVER.FLVCACHE_MAXSIZE=2000

Increased the cache size only delayed the problem from a few hours to 1 day.  It seems that FMS 4.5 is filling up the cache without cleaning anything.  Note that we are doing a very high number of video recordings.  The only way to solve the issue is to restart the FMS service.

One week ago, we were running FMS 3.5.7 on Windows 2003 and we never had this issue.  FMS was running with 500Mb of cache and was working fine for years.  This bugs has appeared since we moved to the FMS 4.5 on a Windows 2008 server.

Coincidentally, Adobe released the 4.5.3 patch that includes fix for memory problems one week ago.  But even this version has the issue.

I saw a similar thread here: http://forums.adobe.com/thread/913107

Anyone can point to a solution?  This is a major bug.

Views

4.8K

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Sep 05, 2012 Sep 05, 2012

Copy link to clipboard

Copied

Hi,

Please go through this piece of information and configure the parameters accordingly:

http://help.adobe.com/en_US/flashmediaserver/configadmin/WS5b3ccc516d4fbf351e63e3d119f2925e64-7ff0.h...

Also, let me know:

     1.     How many streams are being published onto the server?

     2.     What is the container and codec for the published streams?

Thanks,

Shiven

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Sep 05, 2012 Sep 05, 2012

Copy link to clipboard

Copied

Hello Shiven,

thank you for the reply.  A few days back, we increased SERVER.FLVCACHE_MAXSIZE to 2500, and also found that we can clean up the memory faster by setting FLVCachePurge to 1.  We also figured that the old FMS 3.5.7 probably would have had the same issue.  In fact, we saw that the same week we changed to our new server, we also had an increase in usage.

However, this still doesn't explain that the memory fills up so quickly.  It seems that recording videos is a very heavy memory thing for FMS.  Here are the answers to your questions:

1) we had about 35,000 streams on a day that the cache filled up

2) we use standard H.263 in FLV container with NellyMoser mono.

We actually were able to reproduce the problem on any FMS version by setting SERVER.FLVCACHE_MAXSIZE to 1 and doing a few recordings.  After about 5 recordings (of 30 seconds each), the cache problem appears.  By setting the FLVCachePurge to 1, it's harder to reproduce as the cache is cleaned faster.  However, I would expect a cache system to clean up the oldest stream recorded no matter if the caching period is over or not.  There is certainly a major issue to look at here for your team, as we can't expect to just increase the cache size each time the problem re-appear.

Currently, in addition to the two changes that we did in the configuration, we also do a reboot of the FMS service every night.  This is a temporary measure and we need a real solution to this issue.  We can't be rebooting FMS as a final solution.

Ask your team to test and reproduce the cache problem following these steps:

a) set SERVER.FLVCACHE_MAXSIZE=1 in fms.ini

b) record a few stream to the FMS, each of about 30 seconds

c) notice that the cache generate the error and that it's impossible to record another stream until the FMS service is rebooted

EXPECTED behavior: clear the oldest stream, no matter the time set in FLVCachePurge

Please follow-up on this thread with more explaination.

Thank you.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Oct 17, 2012 Oct 17, 2012

Copy link to clipboard

Copied

Hi Nebbb,

Did you manage to get this problem under control? We're seeing exactly the same problem after migrating to 4.5, and tweaking those settings has not made it go away.

Best regards,

Jorge

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Oct 18, 2012 Oct 18, 2012

Copy link to clipboard

Copied

No we did not.  We are rebooting the service daily as the solution.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Oct 22, 2012 Oct 22, 2012

Copy link to clipboard

Copied

What a PITA. Good job on supporting your clients Adobe!

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Mar 11, 2014 Mar 11, 2014

Copy link to clipboard

Copied

I'm having this problem with v5.0.3

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Jul 10, 2014 Jul 10, 2014

Copy link to clipboard

Copied

Hello Shiven,

we reported very specific repro steps more than 2 years ago about this issue.  Others are also getting it in pretty much all version of FMS/AMS.  Can you tell us what is causing this?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Oct 27, 2014 Oct 27, 2014

Copy link to clipboard

Copied

It's now coming on the end of October. Any answer? @ShivenK?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Contributor ,
Nov 10, 2014 Nov 10, 2014

Copy link to clipboard

Copied

My friend Jorge asked me to post this reply as a potential solution.

This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

CONFIG FILE Server.xml

line 251:    <AutoCloseIdleClients enable="true">

line 264:    <FLVCachePurge>10</FLVCachePurge>

line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

line 317:    <ApplicationGC>1</ApplicationGC>

line 319:    <FLVCacheSize>25</FLVCacheSize>

line 594:    <CoreGC>30</CoreGC>

CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

line 74:        SERVER.FLVCACHE_MAXSIZE=3000

NOTE: This value (in MB) should be no more than 25% of total RAM.

3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

Bottom line: allow a core to go idle so that it unloads and releases memory.

Hope that helps.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Contributor ,
Nov 10, 2014 Nov 10, 2014

Copy link to clipboard

Copied

My friend Jorge asked me to post this reply as a potential solution.

This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:

1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).

2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):

CONFIG FILE Server.xml

line 251:    <AutoCloseIdleClients enable="true">

line 264:    <FLVCachePurge>10</FLVCachePurge>

line 265:    <FLVCache retryOnCacheFull="false"> (add attribute to tag)

line 317:    <ApplicationGC>1</ApplicationGC>

line 319:    <FLVCacheSize>25</FLVCacheSize>

line 594:    <CoreGC>30</CoreGC>

CONFIG FILE fms.ini     (renamed to ams.ini in version 5 of FMS)

line 74:        SERVER.FLVCACHE_MAXSIZE=3000

NOTE: This value (in MB) should be no more than 25% of total RAM.

3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:

sync; echo 3 | sudo tee /proc/sys/vm/drop_caches

Bottom line: allow a core to go idle so that it unloads and releases memory.

Hope that helps.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Adobe Employee ,
Nov 10, 2014 Nov 10, 2014

Copy link to clipboard

Copied

Also you can configure Rollover for cores in Application.xml...See

    <LifeTime>

      <RollOver></RollOver>

      <MaxCores></MaxCores>

    </LifeTime>

in Application.xml

Also http://www.adobe.com/devnet/adobe-media-server/articles/hardening_guide.html

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Nov 10, 2014 Nov 10, 2014

Copy link to clipboard

Copied

LATEST

The sync; echo 3 | sudo tee /proc/sys/vm/drop_caches is great for linux host.  Any recommendations for Windows 2008 based machines?   The only solution I foresee would be to have some script to reboot the service daily.  Which is not an option for us with multiple clients globally hitting the system.  

Would load balancing the solution in order to allow idle time for one server?

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines