FMS Server cuts off audio recordings 15-30% of the time after FMS 2.x -> FMS 4.5 upgrade
We have a system where agents connect to record themselves reading a script (audio/microphone only, NO VIDEO) so it can be analyzed for phonetic fluency to assist in hiring those with good English ability.
This system has run for about 6 years on Windows Server 2003 w/ MacroMedia Flash Media Server 2.x without any recording issues.
Recently we had to eliminate Windows Server 2003 from our datacenters and needed to migrate to a new set of machines. We migrated to FMS 4.5 on Windows Server 2008. We have not made any changes to the client-side .SWF code that performs the recording since this product was first deployed.
We are having a problem where 15-30% of recording attempts get cut off after about 15 seconds even though the flash applet indicates recording is still continuing with our microphone level indicator. Under normal circumstances, these recording are usually about 1-2 minutes long. It should be noted that files don't seem to get cutoff less than 15 seconds in, but sometimes it goes farther into the recording past 15-30 seconds before it cuts off.
I have looked through all the config files and couldn't find any setting materially different from the older server or anything that looked like it would make the connections more resilient to flaky network issues.
I have pasted below the log entries from core.log and access.log that correlate to a successful recording and a cut off recording. I tried to search on the (e)2611423 and couldn't find anything actionable.
What can I do here to provide more information or attempt configuration changes to resolve this?
Here is a core.log entry for a normal successful recording:
2016-06-15 12:51:21 5380 (i)2611173 Metadata not available before the flush of first segment. Default metadata recorded for stream E:\LA_ELINK\recording\nae\73c9c126-b14e-462a-a002-c2015bf5510e -
2016-06-15 12:52:18 5380 (e)2611363 Server detected a backward timestamp from 62462 to 62358 in file: E:\LA_ELINK\recording\nae\73c9c126-b14e-462a-a002-c2015bf5510e -
Here is a core.log entry for a recording that is cut off:
2016-06-14 18:04:44 6124 (i)2611173 Metadata not available before the flush of first segment. Default metadata recorded for stream E:\LA_ELINK\recording\nae\e3cfb998-c122-4e2b-8b3a-194c16e6c098 -
2016-06-14 18:05:00 6124 (e)2611423 Failed to record lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 (Unknown Error). -
Here is a access.log entry for a normal successful recording:
connect-pending session 2016-06-15 12:50:45 Eastern Daylight Time 166.137.96.142 172.32.0.11 5380 0 18 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 0 100 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 3073 3073 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
connect session 2016-06-15 12:50:45 Eastern Daylight Time 166.137.96.142 172.32.0.11 5380 0 18 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 0 200 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 3073 3073 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
publish stream 2016-06-15 12:51:15 Eastern Daylight Time lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e 172.32.0.11 5380 0 18 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 0 200 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 3194 3448 normal lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e - - 0 0 - - 0 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
record stream 2016-06-15 12:51:15 Eastern Daylight Time lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e 172.32.0.11 5380 0 18 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 0 200 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 3707 3668 normal lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e E:\LA_ELINK\recording\nae\73c9c126-b14e-462a-a002-c2015bf5510e.flv flv 13 0.000000 - - 513 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
unpublish stream 2016-06-15 12:52:19 Eastern Daylight Time lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e 172.32.0.11 5380 0 19 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 64 200 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 693716 4067 normal lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/73c9c126-b14e-462a-a002-c2015bf5510e - - 0 0 - - 690498 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
disconnect session 2016-06-15 12:52:21 Eastern Daylight Time 166.137.96.142 172.32.0.11 5380 0 18 _defaultRoot_ _defaultVHost_ nexidiaweb _definst_ 96 200 166.137.96.142 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1468774285250 WIN 20,0,0,272 4702111234659533167 693750 4277 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
Here is an access.log entry for the cut off file:
connect-pending session 2016-06-14 18:04:30 Eastern Daylight Time 67.211.2.160 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 0 100 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 3073 3073 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
connect session 2016-06-14 18:04:30 Eastern Daylight Time 67.211.2.160 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 0 200 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 3073 3073 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
publish stream 2016-06-14 18:04:39 Eastern Daylight Time lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 0 200 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 3194 3448 normal lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 - - 0 0 - - 0 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
record stream 2016-06-14 18:04:39 Eastern Daylight Time lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 0 200 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 3707 3864 normal lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 E:\LA_ELINK\recording\nae\e3cfb998-c122-4e2b-8b3a-194c16e6c098.flv flv 13 0.000000 - - 513 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
unpublish stream 2016-06-14 18:05:42 Eastern Daylight Time lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 63 200 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 698846 4495 normal lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 - - rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 rtmp://(redactedserverhostname)/nexidiaWeb/lamedia_elink/recording/nae/e3cfb998-c122-4e2b-8b3a-194c16e6c098 - - 0 0 - - 695628 0 - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx -1 -1.000000 -
disconnect session 2016-06-14 18:05:44 Eastern Daylight Time 67.211.2.160 172.32.0.11 6124 0 17 _defaultRoot_ _defaultVHost_ nexidiaWeb _definst_ 74 200 67.211.2.160 rtmp - rtmp://(redactedserverhostname)/nexidiaWeb rtmp://(redactedserverhostname)/nexidiaWeb - https://(redactedappserverhostname)/nexidia/flash/NexidiaStreamRecorder.swf?version=1465941870018 WIN 21,0,0,213 4702111234860859759 698880 4705 normal - - - - - - - - - - - - - - - - - - - - - - - - - - https://(redactedappserverhostname)/nexidia/default.aspx - - -
