Skip to main content
Anonymous
May 13, 2008
Question

Admin Command Timeout

  • May 13, 2008
  • 3 replies
  • 894 views
We are regularly (once per day minimum) seeing our admin.log fill with error messages like:

Admin Command Timeout: (Command (getSharedObjects) timed out. No response from core(s) : 1796 (active))
Admin Command Timeout: (Command (getInstanceStats) timed out. No response from core(s) : 1796 (active))
Admin Command Timeout: (Command (getActiveInstances) timed out. No response from core(s) : 1796 (active))

These are in response to calls we make to admin to get stats for an instance. We would normally have 100 instances running and make call to admin once per instance every thirty seconds, which doesn't seem to me be particularly heavy usage (?).

When this happens the FMS server starts to refuse connections (but not always, it appears random which are allowed through) and doesn't free up connections (we see the client count continually creep up until it is in the 100's) even though there is no one using the app at all. Nothing can restore normal functioning except restarting the FMS service. The problem occurs at random times, sometimes after just an hour of use, other times it can run for 24 hours or more without problem but always it ultimately fails.


Setup:

Windows 2003
Flash Media Interactive Server 3.0.1 r123
This is not a load problem as at peak the server is using 15% CPU and about 30% of available memory and there is plenty of free disk space.
It happens on 3 different servers I have also seen it happen when only one webcam is being streamed.

I would be very grateful for any pointers anyone can give as this is making FMS unusable for us and we are already looking around for more stable alternatives.

Thanks

David
    This topic is closed to new replies. Start a new post to keep the conversation going.

    3 replies

    Kristian Wright
    Kristian Wright
    Known Participant
    June 4, 2008
    Hi.

    I've notices similar issues, and looking into the Windows task manager, it seems that one of the FMS cores are using up a stack of memory! When stopping this core from task manager, it seems to free all of the connections to the instances.

    So I'm not sure that it's a problem with the admin, but maybe with the core itself...

    Have you come across any solutions or advice on this at all?
    Anonymous
    May 15, 2008
    "we see the client count continually creep up until it is in the 100's) even though there is no one using the app at all. "

    Thats strange and shouldn't happen; You might want to try tracing the IP address and HTML/HTML referrer domains of the clients to track down those hidden clients :)

    Not much sure about "Admin Command Timeout" thing;except seeing it rarely under load.
    Anonymous
    May 18, 2008
    Well, really, none of these problems should happen! But, sadly, they do and not just for me.
    A Google search will show that these Admin timeouts are a very common problem but no-one seems to have a solution. More worryingly, I have never seen any suggestion from Adobe about what might cause them.

    Since my original post I have made two changes to our app:
    (1) when the 'owner' of the room leaves I unload the instance (after a 30 seconds delay)
    (2) I amended the application.xml for this app to timeout idle connections after 2 minutes (the default is no timeout)

    This has made things slightly better in that the app now runs for 2/3 days without failing (rather than failing twice a day). But, it still fails and the only thing that brings it back is a reboot of the server which is totally unacceptable in a live environment where we usually have 500+ clients watching 100 live streams at any one time (many on a pay-per-minute basis). The only other significant thing I notice now is that once the machine has been running for a couple of days the logging stops completely for some (but not all) instances i.e. nothing is displayed in the admin console and nothing is written to the log files. It's as if the FMS Admin has just given up.

    Re. your suggestion about checking IPs etc. There are no "hidden clients"! When the Admin fails it is possible to be on a machine with a single connection (i.e. my client). When I reload my client there is a trace that I have disconnected, then reconnected again *but* the connection count goes up by one. If I reload twenty times the count will increase to 20. Even if I close my browser the connections will apparently stay, sometimes for days!

    As I mentioned, this happens on multiple servers with lots of different clients and very, very low load. When it happens is, so far, unpredictable other than it will happen and bring down our live system.

    Terms & ConditionsAccessibility statement
    Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices