Health check failed due to server expiry

Report · Jul 02, 2010

Our self-hosted adobe connect server keeps dropping in the middle of the night. Worse, it seems to drop meeting connections regularly. The error logs read as follows:

#Version: 1.0
#Start-Date: 2010-07-02 02:02:56 PDT
#Software: Macromedia Breeze Platform [connect_750_r134]
#Date: 2010-07-02
#Fields: date    time    x-comment    x-module    x-status    x-severity    x-catagory    x-user    x-access-request    time-taken    db-logical-io    db-transaction-update-count
2010-07-02    02:02:56    "Health check failed due to server expiry (expiry date=2010-07-02T02:03:31.327-07:00, current time=2010-07-02T02:03:46.903-07:00)."    PLATFORM_FRAMEWORK    SERVER_EXPIRED    C    S    -    "-"    -    -    -
2010-07-02    02:07:07    "Server connect has been designated the master."    PLATFORM_FRAMEWORK    BECAME_MASTER    I    S    -    "-"    -    -    -
2010-07-02    02:11:36    "Health check failed due to server expiry (expiry date=2010-07-02T02:32:56.403-07:00, current time=2010-07-02T02:33:02.853-07:00)."    PLATFORM_FRAMEWORK    SERVER_EXPIRED    C    S    -    "-"    -    -    -
2010-07-02    02:13:08    "Server connect has been designated the master."    PLATFORM_FRAMEWORK    BECAME_MASTER    I    S    -    "-"    -    -    -
2010-07-02    02:13:55    "Health check failed due to server expiry (expiry date=2010-07-02T02:38:06.463-07:00, current time=2010-07-02T02:38:28.003-07:00)."    PLATFORM_FRAMEWORK    SERVER_EXPIRED    C    S    -    "-"    -    -    -
2010-07-02    02:14:28    "Start up error: jrunx.kernel.ServiceException: The ServletEngineService was not found."    START_UP    START_UP_ERROR    E    S    -    "-"    -    -    -
2010-07-02    02:16:40    "Server connect has been designated the master."    PLATFORM_FRAMEWORK    BECAME_MASTER    I    S    -    "-"    -    -    -
2010-07-02    03:09:09    "Server connect has been designated the master."    PLATFORM_FRAMEWORK    BECAME_MASTER    I    S    -    "-"    -    -    -
2010-07-02    03:09:43    "Health check failed due to server expiry (expiry date=2010-07-02T03:11:14.177-07:00, current time=2010-07-02T03:11:18.673-07:00)."    PLATFORM_FRAMEWORK    SERVER_EXPIRED    C    S    -    "-"    -    -    -
2010-07-02    03:10:31    "Start up error: jrunx.kernel.ServiceException: The ServletEngineService was not found."    START_UP    START_UP_ERROR    E    S    -    "-"    -    -    -
2010-07-02    03:12:19    "Server connect has been designated the master."    PLATFORM_FRAMEWORK    BECAME_MASTER    I    S    -    "-"    -    -    -
2010-07-02    03:13:17    "Health check failed due to server expiry (expiry date=2010-07-02T03:23:32.150-07:00, current time=2010-07-02T03:23:34.507-07:00)."    PLATFORM_FRAMEWORK    SERVER_EXPIRED    C    S    -    "-"    -    -    -
2010-07-02    03:13:59    "Start up error: jrunx.kernel.ServiceException: The ServletEngineService was not found."    START_UP    START_UP_ERROR    E    S    -    "-"    -    -    -

When I log on to the server, it is so slow it is almost unusable, but CPU is hardly being used. I've tried restarting the Adobe Connect service but it has no effect. Typically rebooting the box solves the problem.

This feels suspiciously like a memory leak. What is the "server expiry" all about?

Report · Jul 02, 2010

Additional info from the debug logs. I found this repeated over and over:

[07-02 01:01:30] fmg-health-check (d) FMGManager Performing Health Check

[07-02 01:01:30] fmg-health-check (d) 0ms spid:80 com.macromedia.airspeed.db.SQLServerUtilityQueries.ping

[07-02 01:01:30] fmg-health-check (d) 0ms spid:80 com.macromedia.breeze.fmg.FMGManager.getHealthCheckHostsFromDB(5001)

[07-02 01:01:30] fmg-health-check (d) FMGManager Performing Health Check. Host count: 1

[07-02 01:01:30] fmg-health-check (d) FMGManager Firing health check request with URL: http://xxx.xxx.edu:2222/admin/getFMGStatus?auser=sa&apswd=BtfZgecLpAHIJE68u+waPg==

[07-02 01:01:32] fmg-health-check (d) Connection refused: connect

Report · Sep 12, 2010

This error is also happening to me. What can be?

Help

Report · Sep 16, 2010

I'm glad you revived this thread. I didn't realize I had left it unresolved.

The problem is fixed for us. We inadvertently made two changes simultaneously, so we're not sure which solved the problem. I'll explain both.

Our Sr. LAN Admin got in touch with Adobe Tech Support. The Adobe tech reindexed our Connect database and removed a column that he said was causing a problem. I cannot find any specifics in the email about which column was removed, or what problem the column was causing.

We are running Adobe Connect on a VMWare virtual server. While the Senior LAN Admin was working with Adobe, Our VMWare specialist made some configuration changes. He reduced the number of CPUs from 4 to 1, and reduced the memory from 8gb to 3gb.

When everyone was done, we rebooted the machine and the problem was resolved.