Skip to main content
A
Anonymous
April 22, 2010
Question

FLVCache segment is full

  • April 22, 2010
  • 1 reply
  • 1266 views

Hi everybody.


I'm using FMS 3.5.3 (with updates) on my RedHat pool server.


Let me introduce (i cant tell much more details about this because of privacy).

Ive got 10 real machines under an hardware front VIP router. It just distribute the requests between these 10 machines.

Each machine has 4 Quadcore processor, with 8GB ram (it isnt so much, but a week ago we had 4GB by peer). Each machine has an great MD3000I storage that serves the mp4 files as audio streaming.


So far, so good...


And now here I'll tell you the real issues about my problems.

I've got an application that the users (from internet) chooses the audio wich they want to listen to.
There are more than 200.000 files(and growing) to be streamed to the internet clients, so I just use Stream from "recorded" files.

These huge amount of files are separated into 2 "applications", and these applications loads 3 instances as I can see in <Distribute numprocs="3">Instances</Distribute> in the Application.xml config file.

In my "fms.ini" file, the parameter SERVER.FLVCACHE_MAXSIZE was with default value(500MB) and i started notice in my core.00.log file, that I took a lot of "FLVCache segment is full" message.
I realized that each time I took this message, the streaming stops to some client and the client could be me either.
I cant tell about my number of clients that connects to my servers, but its a huge number of clients, streaming 1 or more files of my collection of more than 200.000 mp4 files.
There are peeks during the day that My servers generates lots(Kind of 90.000 each day) of "FLVCache segment is full" message.
I raise the value to 3000MB and my server crashed down seconds after some connections. Later I've tried again with 1000 so it crashed again. Finnaly I put 750 and it could handle and the Messages of Flvcache full went down to some acceptable(1000 each day).

I want to solve this issue of "FLVCache segment full"...

My doubts:
- FLVCache value is 500 for example.
  1 - I'll take one instance per aplication, so in my case (ive got 2 apps) it will take 2 processes, and these processes will use NUMBER_OF_PROCESSES * FLVCACHE_MAXSIZE = TOTAL_MEMORY of fmscore process?
  2 - Each application will have his own FLVCache of 500Mb?
  3 - If i raise the number of applications, my memory will be splitted in more spaces so my problems will raise?
  4 - If i put every files to be served into just one application, with 8 instances (like my number of processors), can I raise the value of FLVCACHE to 3000MB so I'll have more memory to cache works with more space and will stop the errors?

Ive read all the docs, and didnt figured out in which level FLVCache would act, and how it breaks my files.
5 - Can anyone explain how my mp4 files are cached and threated? There are any parameters to threat that?

7 - My IO is a kind of fast comparing to usual... Can I do more IO and serve the files direct from disk or a kind of virtual memory to FMS work?

I cant show more details from my kind of application, but if i was unclear about my issues and explanations, ask me to explain in other words...
English isnt my natural language! 🙂

Tks!
Everton

    This topic has been closed for replies.

    1 reply

    S
    SE_0208
    Participating Frequently
    April 23, 2010

    To start with we generally configure FLVCache to be 1/4th of RAM size.But I am no way saying that's best or optimal setting.

    These huge amount of files are separated into 2 "applications", and these applications loads 3 instances as I can see in <Distribute numprocs="3">Instances</Distribute> in the Application.xml config file.

    Did not get what you mean.I think there is slight misunderstaning you have regarding this. The setting which you have shown here decided number of core process which can come up and not number of instances of application. Loading of instance of application depends upon your connection URI (which instance you connect to)

    I raise the value to 3000MB and my server crashed down seconds after some connections. Later I've tried again with 1000 so it crashed again. Finnaly I put 750 and it could handle and the Messages of Flvcache full went down to some acceptable(1000 each day).

    This might be because you tried configuring FLVCache much higher than what can ideally work. Any linux process can access max 3GB of address space so probabaly that explains crash when you set it to 3000MB. But 1000MB should have worked ideally.

      1 - I'll take one instance per aplication, so in my case (ive got 2 apps) it will take 2 processes, and these processes will use NUMBER_OF_PROCESSES * FLVCACHE_MAXSIZE = TOTAL_MEMORY of fmscore process?

    [SE_0208] - There is no guarantee that it will get spread across 2 core process just because  you have two apps. Also [NUMBER_OF_PROCESSES * FLVCACHE_MAXSIZE = TOTAL_MEMORY of fmscore process], does not make such sense as FLVCACHE_MAXSIZE is per process. Now if you meant TOTAL_MEMORY for all fmscore processes then again it won't be true as fmscore migth take memory apart from FLVCache for doing some other tasks.


      2 - Each application will have his own FLVCache of 500Mb?

    [SE_0208] - Nope,its per fmscore process


      3 - If i raise the number of applications, my memory will be splitted in more spaces so my problems will raise?

    [SE_0208] - What do you mean by memory getting split?


      4 - If i put every files to be served into just one application, with 8 instances (like my number of processors), can I raise the value of FLVCACHE to 3000MB so I'll have more memory to cache works with more space and will stop the errors?

    Ive read all the docs, and didnt figured out in which level FLVCache would act, and how it breaks my files.

    [SE_0208] - To be frank i did not get what exactly you meant here. I would suggest , make numprocs to 1 and try 2000MB setting and see if that works.


    5 - Can anyone explain how my mp4 files are cached and threated? There are any parameters to threat that?

    [SE_0208] - Again what do you mean by "threated" ? Did you mean treated . Caching works for all formats, i know name might be confusing but when FMS started there was only format and that was FLV , hence the name. Whenever you play any file, it gets cached in FLVCache whether its flv,mp4 or mp3.

    7 - My IO is a kind of fast comparing to usual... Can I do more IO and serve the files direct from disk or a kind of virtual memory to FMS work?

    [SE_0208] - Not so sure about that.

    Also would like to know , are you trying edge-origin set-up?
    Let me know if still you need any more help?

    A
    Anonymous
    April 23, 2010

    Let me show more detailed configuration.

    I have more than 200k mp4 files to stream.

    The configuration is pointing like this:

    my.domain.name/appA/

    Under this url/virtual host there is for example 100k mp4 files...

    my.domain.name/appB/<the_path_to_my_file>/filename.mp4

    Under this url/virtual host there is for example the another 100k mp4 files

    In this case, my configs are distributed into 1 process by application, so i'll get 2 coreprocess to each of appA and appB, right? And there are the base processes: 1 mastercore, 1 edge, 1 admin, 1 for appA and 1 for appB. This is what i get with "ps -ef | grep adobe"

    I'll try this option and let you know. Cause i'm using 3 processes per application.

    The split question was like this:

    Each of my mp4 file has almost 1.5MB. My question is: streaming 10 mp4 files with that same size (1.5Mb) will take from my FLVCache 15Mb? I was wondering if I could change (if this is true).

    I'm streaming simultaneously by machine, 9k files at some peaks... So my FLVCache is really overloaded.

    Thanks a lot by the responses!

    Ill try another cases!

    I feel like i'm trying to unlok an safe.

    A kind of open source would be great to know this kind of stuff more quick...

    Edge and origin is in the same machine. Would you configure another way?

    Like 2 machines from my pool being origin and other 8 for example being the edge?

    I could do this too in an second time, cause my fiber network among these servers are sleeping. As my CPUs and my IO...All really sleeping... But I cant tell the same about memory!!! Its really getting swet! :-)

    Thanks a lot!

    S
    SE_0208
    Participating Frequently
    April 26, 2010

    everton.rosario wrote:

    Let me show more detailed configuration.

    I have more than 200k mp4 files to stream.

    The configuration is pointing like this:

    my.domain.name/appA/

    Under this url/virtual host there is for example 100k mp4 files...

    my.domain.name/appB/<the_path_to_my_file>/filename.mp4

    Under this url/virtual host there is for example the another 100k mp4 files

    In this case, my configs are distributed into 1 process by application, so i'll get 2 coreprocess to each of appA and appB, right? And there are the base processes: 1 mastercore, 1 edge, 1 admin, 1 for appA and 1 for appB. This is what i get with "ps -ef | grep adobe"

    I'll try this option and let you know. Cause i'm using 3 processes per application.

    [SE_0208] - Are you saying that you have changed scope to "applications" in Application.xml ? How are you concluding that 1 process is for appA and 1 is for appB. Also on what basis are you saying that you are using 3 processes per application. Please do get confused by numProcs = 3 - that no way means each application would use 3 processes. Also are you using more than one vhost?

    The split question was like this:

    Each of my mp4 file has almost 1.5MB. My question is: streaming 10 mp4 files with that same size (1.5Mb) will take from my FLVCache 15Mb? I was wondering if I could change (if this is true).

    I'm streaming simultaneously by machine, 9k files at some peaks... So my FLVCache is really overloaded.

    [SE_0208] - Yes that's more or less true that you will require 15Mb but that does not mean you will keep on increasing your FLVCache, its like when you do not have space on FLVCache , newer data would overwrite older one in FLVCache and if request comes for older data again which is overwritten , it will be fetched from disk.

    Thanks a lot by the responses!

    Ill try another cases!

    I feel like i'm trying to unlok an safe.

    A kind of open source would be great to know this kind of stuff more quick...

    Edge and origin is in the same machine. Would you configure another way?

    Like 2 machines from my pool being origin and other 8 for example being the edge?

    I could do this too in an second time, cause my fiber network among these servers are sleeping. As my CPUs and my IO...All really sleeping... But I cant tell the same about memory!!! Its really getting swet! :-)

    [SE_0208] - It does not make great sense to have edge and origin on same machine, to be frank it defeats the whole process of having Edge Server. It would be better like how you said in second like , 2 machines as Origin and 8 machines as Edge and this 8 machine will share the load of all subscribers.

    Thanks a lot!

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices