Skip to main content
D
dejanp777
Inspiring
May 22, 2011
Question

FMIS restarting and memory issue

  • May 22, 2011
  • 3 replies
  • 2244 views

I have Flash Media Interactive Server 3.5.5 on Windows 2008 server. I developed video chat application (many to many), it's only 1 FMS application with about 50-100 room instances and 1 lobby instance to which all users are connected. There are up to 600 - 700 users at a time which produce about 2000-2500 connections to FMS.
Problem I have is with steady raise of memory usage, which if goes little above 2GB will make FMS unresponsive and it would restart itself after about 1-2hours. To avoid that Im restarting FMS manualy every 3-4 days. Interesting thing I noticed is that if I restart FMS from admin console, used RAM memory won't fall at levels as if it would right after whole Windows server is restarted.

In my Application.xml there are following configs:
<Scope></Scope>
<Distribute numprocs="3">Instances</Distribute>

These are things from logs during FMS unresponsive period or right before it:

master.log
2011-05-21    22:01:11    996    (i)2581223    Core (3592) is no longer active.    -
2011-05-21    22:01:16    996    (i)2581223    Core (1868) is no longer active.    -

admin.log
2011-05-21    22:01:06    2832    (i)2581242    Core (3592) disconnected : _defaultRoot_:_defaultVHost_:::_0.    -
2011-05-21    22:01:09    2832    (i)2581242    Core (1868) disconnected : _defaultRoot_:_defaultVHost_:::_1.    -

edge.log
2011-05-21    14:22:23    840    (e)2661034    Connect failed ( , -1 ) : Connect failed: (10061)    -

2011-05-21    14:23:05    840    (e)2661034    Connect failed ( , -1 ) : Connect failed: (10061)    -

2011-05-21    22:01:06    840    (i)2581250    Edge disconnected from core (3592).    -
2011-05-21    22:01:09    840    (i)2581250    Edge disconnected from core (1868).    -

access.log
session    disconnect    2011-05-21    21:29:06    6632    68.191.113.251    29476    39743    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    72.198.193.1    4759    6837    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    83.70.249.91    11996    14797    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    108.8.209.2    57227    76111    -    -    -    -    200    -
session    disconnect    2011-05-21    21:29:06    6632    212.183.140.17    3073    3595    -    -    -    -    200    -

Please help me with this issue.

    This topic has been closed for replies.

    3 replies

    D
    dejanp777Author
    Inspiring
    July 29, 2011

    Latest problem that I have more often is that Windows 2008 where FMS 3.5 is stops working, so manual restart of Windows 2008 is required. In FMS logs I don't see any new items that I haven't wrote in previous posts here. Only thing I noticed is that in _definst_ logs there is massive disconnection of all users in 1-2 seconds when this happens. And this problem happens when there is big number of users connected, which is around 600 which have around 2000-2500 connections to FMS.

    Thing that I suspect could be issue is that my Application Process Scope is set to default vhost, so whole application with all of it's instances is set in 1 Windows process which is sometimes 220-250MB large. Do you think I should set Application Process Scope to inst to avoid this? Although I have up to 60 application instances running at same time.

    Please help me with this.

    Nikhil_Kalyan
    Nikhil_Kalyan
    Participating Frequently
    July 29, 2011

    Hi,

    Yes changing the process scope can enhance the performance , but make sure your work flow is intact while distributing the traffic among different processes. These processes will not be able to share resoruces (read, media and objects) and hence is different.


    Regarding the total load on the server, i think it would still have the same (more or less) memory usage, that is now distributed on many processes, right ?

    D
    dejanp777Author
    Inspiring
    June 2, 2011

    Hi,
    Sorry for delay in reply.
    After FMS restart, memory falls but not in levels where it's after Windows restart, even when there are no existing users who are attempting reconnection.

    In core logs I can only find things like:
    2011-06-01    19:17:12    5112    (e)2631029    Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 75.205.202.118, App: chatcity/_definst_, Protocol: rtmp, Client: 7011883898884410480, Handle: 2265252077) : D5 D9 BE F8 ; 1F 82 p 3 , $ B8 8D S J ^ ) 94 FD S 0B 81 80 $ FF A9 > 83 CD 87 A8 L H F4 9D     -


    2011-06-01    21:34:25    5112    (e)2611392    Exception while processing message: Unexpected virtual message from client 6140488000333434699: DD 82 . 94 EB FC i 00 ( C5 E6   


    2011-05-28    11:26:59    3272    (i)2581368    Client 6080275479418588227 has been idle for 36012 seconds.    -


    In configurations I have these settings (just to repeat that application is video chat, so there are no FLV files involved, only live streaming):
    <FLVCacheSize>10</FLVCacheSize>

    <EnableAggMsgs>false</EnableAggMsgs>

    <AutoCloseIdleClients enable="true">
    <CheckInterval>600</CheckInterval>

    <Queue enabled="false">


    Server specification is:
    Intel Xeon X5450 @ 3.00GHz (2 processors)
    16GB RAM

    If there is anything more needed please tell me.

    Nikhil_Kalyan
    Nikhil_Kalyan
    Participating Frequently
    June 3, 2011

    Hi,

    Thanks for the information.

    When you said the memorny is still not less, you are talking of the memory consumed by FMS processes ?

    Once the FMS service (and the FMS admin service) is restarted, the old processes are shutdown and new ones come up. Can you individually record the memory usage of these processes ? Probably there is something else that's blocking the memory to leak. Make sure the old Fms processes are properly shutdown during the restart (kill if not clean).

    Regarding the bad network data messages in your core logs, they come when there are 'unwanted' requests coming to FMS. For example, if there is a http request which is corrput that is sent on to the FMS edge process (which is expecting rtmp and when http is disabled), it might reject this request and log a message something very similar to what you are seeing. I have found these messages often when the machine is sort of virus affected also. Please check for these things as well.

    I hope the disabling of queue and aggre messages is to support your video chat application, the settings look fine to me and hence not a problem.

    D
    dejanp777Author
    Inspiring
    July 11, 2011

    Thanks for your answers, I'm replying with delay because I wanted to check these things some more.

    When I said 'memory is not less', I meant that total memory in Windows after FMS restart doesn't return to memory levels when Windows is restarted, but I guess that's because of some Windows processes and not connected to FMS.


    As for FMS processes, these are the things are notice, few hours after FMS restart I get 1 FMSCore.exe that is around 200-220MB, and 3-4 FMSCore's that are around 40-70MB. Size of that largest FMSCore.exe doesn't go down even when there are much less users connected to FMS. And like I said before, after 3 or more days of running FMS becomes unresponsive, and restarts itself. Right after FMS restart largest FMSCore.exe weights around 70MB but 15mins after it's already 120MB as users connect to video chat application.

    As for logs, my core.log files have several lines like:


    2011-07-04    03:59:24    3308    (e)2611392    Exception while processing message: Unexpected virtual message from client 6446421578071032403: 93 { # 92 " : t 00 00 00 02     -
    2011-07-04    03:59:24    3308    (e)2611029    Bad network data; terminating connection : chunkstream error:message length 12086012 is longerthan max rtmp packet length    -
    2011-07-04    03:59:24    3308    (e)2631029    Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 94.4.57.121, App: chatcity/stream, Protocol: rtmp, Client: 6446421578071032403, Handle: 2497053121) : 88 1E & D7 FB / 8D 7 99 CB X ...

    2011-07-02    14:09:57    1228    (e)2611363    Server detected a backward timestamp from 7990 to 7985 in file: C:\Program Files (x86)\Adobe\Flash Media Server 3.5\applications\chatcity\streams\_definst_\captured_16627033    -

    While edge.log has:

    2011-07-04    04:58:56    1384    (e)2661034    Connect failed ( , 8134 ) : Connect failed: (10061)    -
    2011-07-02    21:09:05    6864    (e)2661034    Connect failed ( , -1 ) : Connect failed: (10061)    -

    And master.log has:

    2011-07-04    00:30:27    7128    (i)2581223    Core (6280) is no longer active.    -
    2011-07-04    00:30:27    7128    (i)2581223    Core (4352) is no longer active.    -

    Nikhil_Kalyan
    Nikhil_Kalyan
    Participating Frequently
    May 24, 2011

    Hi,

    Thanks for trying out FMS.

    When you restart the FMS, one reason why the memory is still high is because of the existing users. They might as well attempt reconnection as soon as the FMS is up again, retaining your memory usage. There must be a sudden fall and rise if that is the case. Please let us know if this is correct.

    You didn't mention the core logs, are there any error statements in the core log when it dies ?

    I would also like to know the other configurations for FMS like FLV cache size , aggregate messages (enabled/disabled), queue enabled/disabled etc. Can you please provide those details as well ?


    What is the system configuration of the machine on which the FMS is installed ?

    Thank you !

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices