Copy link to clipboard
Copied
We just migrated our service to FMS 4.5. Since our new server is in production, we started having the following error message in the core.log files:
The FLV segment cache is full
There is very little information about this problem on the web. The only information found is to increase the value of parameter SERVER.FLVCACHE_MAXSIZE in the fms.ini file. We tried this:
SERVER.FLVCACHE_MAXSIZE=2000
Increased the cache size only delayed the problem from a few hours to 1 day. It seems that FMS 4.5 is filling up the cache without cleaning anything. Note that we are doing a very high number of video recordings. The only way to solve the issue is to restart the FMS service.
One week ago, we were running FMS 3.5.7 on Windows 2003 and we never had this issue. FMS was running with 500Mb of cache and was working fine for years. This bugs has appeared since we moved to the FMS 4.5 on a Windows 2008 server.
Coincidentally, Adobe released the 4.5.3 patch that includes fix for memory problems one week ago. But even this version has the issue.
I saw a similar thread here: http://forums.adobe.com/thread/913107
Anyone can point to a solution? This is a major bug.
Copy link to clipboard
Copied
Hi,
Please go through this piece of information and configure the parameters accordingly:
Also, let me know:
1. How many streams are being published onto the server?
2. What is the container and codec for the published streams?
Thanks,
Shiven
Copy link to clipboard
Copied
Hello Shiven,
thank you for the reply. A few days back, we increased SERVER.FLVCACHE_MAXSIZE to 2500, and also found that we can clean up the memory faster by setting FLVCachePurge to 1. We also figured that the old FMS 3.5.7 probably would have had the same issue. In fact, we saw that the same week we changed to our new server, we also had an increase in usage.
However, this still doesn't explain that the memory fills up so quickly. It seems that recording videos is a very heavy memory thing for FMS. Here are the answers to your questions:
1) we had about 35,000 streams on a day that the cache filled up
2) we use standard H.263 in FLV container with NellyMoser mono.
We actually were able to reproduce the problem on any FMS version by setting SERVER.FLVCACHE_MAXSIZE to 1 and doing a few recordings. After about 5 recordings (of 30 seconds each), the cache problem appears. By setting the FLVCachePurge to 1, it's harder to reproduce as the cache is cleaned faster. However, I would expect a cache system to clean up the oldest stream recorded no matter if the caching period is over or not. There is certainly a major issue to look at here for your team, as we can't expect to just increase the cache size each time the problem re-appear.
Currently, in addition to the two changes that we did in the configuration, we also do a reboot of the FMS service every night. This is a temporary measure and we need a real solution to this issue. We can't be rebooting FMS as a final solution.
Ask your team to test and reproduce the cache problem following these steps:
a) set SERVER.FLVCACHE_MAXSIZE=1 in fms.ini
b) record a few stream to the FMS, each of about 30 seconds
c) notice that the cache generate the error and that it's impossible to record another stream until the FMS service is rebooted
EXPECTED behavior: clear the oldest stream, no matter the time set in FLVCachePurge
Please follow-up on this thread with more explaination.
Thank you.
Copy link to clipboard
Copied
Hi Nebbb,
Did you manage to get this problem under control? We're seeing exactly the same problem after migrating to 4.5, and tweaking those settings has not made it go away.
Best regards,
Jorge
Copy link to clipboard
Copied
No we did not. We are rebooting the service daily as the solution.
Copy link to clipboard
Copied
What a PITA. Good job on supporting your clients Adobe!
Copy link to clipboard
Copied
I'm having this problem with v5.0.3
Copy link to clipboard
Copied
Hello Shiven,
we reported very specific repro steps more than 2 years ago about this issue. Others are also getting it in pretty much all version of FMS/AMS. Can you tell us what is causing this?
Copy link to clipboard
Copied
It's now coming on the end of October. Any answer? @ShivenK?
Copy link to clipboard
Copied
My friend Jorge asked me to post this reply as a potential solution.
This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:
1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).
2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):
CONFIG FILE Server.xml
line 251: <AutoCloseIdleClients enable="true">
line 264: <FLVCachePurge>10</FLVCachePurge>
line 265: <FLVCache retryOnCacheFull="false"> (add attribute to tag)
line 317: <ApplicationGC>1</ApplicationGC>
line 319: <FLVCacheSize>25</FLVCacheSize>
line 594: <CoreGC>30</CoreGC>
CONFIG FILE fms.ini (renamed to ams.ini in version 5 of FMS)
line 74: SERVER.FLVCACHE_MAXSIZE=3000
NOTE: This value (in MB) should be no more than 25% of total RAM.
3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Bottom line: allow a core to go idle so that it unloads and releases memory.
Hope that helps.
Copy link to clipboard
Copied
My friend Jorge asked me to post this reply as a potential solution.
This is the result of one or more core(s) grabbing memory and then not letting go. Eventually, the core crashes. We solved this by:
1) making sure that instances don't stay live indefinitely (that prevents the core it sits on from unloading).
2) implementing the following configs so that idle instances & cores unload more quickly (ignore the line numbers- just search for the tags):
CONFIG FILE Server.xml
line 251: <AutoCloseIdleClients enable="true">
line 264: <FLVCachePurge>10</FLVCachePurge>
line 265: <FLVCache retryOnCacheFull="false"> (add attribute to tag)
line 317: <ApplicationGC>1</ApplicationGC>
line 319: <FLVCacheSize>25</FLVCacheSize>
line 594: <CoreGC>30</CoreGC>
CONFIG FILE fms.ini (renamed to ams.ini in version 5 of FMS)
line 74: SERVER.FLVCACHE_MAXSIZE=3000
NOTE: This value (in MB) should be no more than 25% of total RAM.
3) running the following cron daily on linux (during low/off peak hours) in order to clear the page cache:
sync; echo 3 | sudo tee /proc/sys/vm/drop_caches
Bottom line: allow a core to go idle so that it unloads and releases memory.
Hope that helps.
Copy link to clipboard
Copied
Also you can configure Rollover for cores in Application.xml...See
<LifeTime>
<RollOver></RollOver>
<MaxCores></MaxCores>
</LifeTime>
in Application.xml
Also http://www.adobe.com/devnet/adobe-media-server/articles/hardening_guide.html
Copy link to clipboard
Copied
The sync; echo 3 | sudo tee /proc/sys/vm/drop_caches is great for linux host. Any recommendations for Windows 2008 based machines? The only solution I foresee would be to have some script to reboot the service daily. Which is not an option for us with multiple clients globally hitting the system.
Would load balancing the solution in order to allow idle time for one server?