Known Participant

Question

FMS 3.5.3 does not discard clients, streaming stops working...

Forum|Forum|16 years ago
April 29, 2010
5 replies
25702 views

We have a FMS 3.5.3 running on a production server, a 2x quad-core Xeon E5345 CPU based hardware with Windows Server 2008 R2 64-bit OS.

The FMS service streams live streams to hundreds of simultaneous Adobe Flash Player 10 clients through RTMP at port 1935.

Occasionally, usually with a higher number of simultaneous clients connected (around 1, one can see a rising number of clients of the loaded (single) application (never diminishing) through the Administration Console. It appears that the clients "linger" - i.e. the number only counts up, never down, from then on.

As if this was not sign of problems enough in itself, the bandwidth graph drops to near zero - i.e. it appears that streaming stops working too. Restarting application helps. The logs do not show anything UNTIL AFTER application is attempted unloaded via console. There are messages of various importance in the event log, among these the weird warning "Asynchronous I/O operation failed (Failed to attach to completion port: The parameter is incorrect. 87)." There is more too, which I can provide on demand.

The application is pretty rudimentary. Of the entire API, we only use Application, Client and SharedObject classes. There is a single persistent shared object, and that's it.

We are using a clean install. Only Microsoft supplied and FMS processes are running. Windows Firewall is on though, and has rules for letting in connections on ports 1935 and 1111.

I am lost here. We need a stable server, and I have to restart it manually every day, sometimes twice in an hour, which frustrates me.

I have seen that there are others with a similiar problem. Has there been any solution for this?

This topic has been closed for replies.

A

Anonymous

Sure :

Event Type:    Error
Event Source:    FMS (Core)
Event Category:    (263)
Event ID:    1029
Date:        8/26/2010
Time:        3:53:30 PM
User:        N/A
Computer:    XYZ
Description:
Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 90.19.24.235, App: LiveVideoStream3/400009843, Protocol: rtmp, Client: 5283135158007779696, Handle: 2416903194) : 0D 07 EC a F7 C6 G 97 80 I a F4 X 0D 03 D9 E8 4 1F 91   12 EE y I A 9 A7 81 A0 { ) 81 9C 0 04 BC A8 # q C0 0 0D C3 E0 > 10 ? Y T t 8C 8E C5 > FA A0 3 0F B BD E9 4 80 F4 e 1A 0D 07 EC P   8F 81 80 87 00 F5 ` 15 - 04 F7 DA E 03 A F9 1F ~ ~ V R 15 F1 F0 A2 @ 8F # 8D 04 A9 F2 E2 E5 R 91 C2 84 . + 8F 3 08 05 E0 86 > F2 EE 88 A6 1E 9D 11 13 88 H C8 ` DE 95 92 C6 15 < 91 E5 - v 98 8B V i E5 AB 90 0 0D   D0 FB C9 BB 03 00 DA ^ 07 C2 0B E4 a C4 H D F5 [ | E5 % $ A4 A2 1E E7 % ! ! 1D 10 F4 x AF 1A u 80 D0 C1 D 06 N D0 1A 18 ( G 91 d O AF 7F } 7F | 7F 9B F3 EB FB F3 E3 FC DF _ 1F _ ^ FF D3 BC AA 03 00 1C 0C 0C - 06 8F F9 E9 ` 18 00 F0 5 F2 7 B3 0F F3 ; : : : 7F z 99 D9 D9 D1 D3 EB DD CE 8B 03 # 92 B0 C9 C6 F1 F6 A5 80 16 v Z 01 g o o FC C4 02 8B 80 , C0 02 8C 00 - DD FA E7 A6 AE > 4 s D3 FD CB 0D C 2 C3 ` CD ED F5 g N 1E 1F C 86 83 E9 AF 16 0C 9E ) A7 F9 89 09 09 09 ? FF A5 . q a p d Y 08 E7 A9 95 96 18 18 C0 BC E4 B4 B0 B0 2 + 0C E0 DD E6 h b n e 0A F1 3 # 13 # h G 9B 98 15 86 F 85 A1 94 + BE 09 9E j F2 c 0 CE 09 E6 S 83 A3 s B7 C2 BB T D1 C6 C5 A7 01 91 d ; B7 5 y 87 9E f 01 f EA 9F 0C EF 82 E0 CC CE | F1 a C8 g 0C F0 E8 A2 06 d G P 3 2 80 17 0C 01 . E7 A1 11 12 80 A D 0C C8 E0 9E 94 17 P 0A < V 0D 00 ) 09 ) E8 3 CF O I F8 4 00 AA 4 2 * A0 14 i F0 h 01 b + 9D 8F AB U T 00 94 0A 0D 00 2 E 06 80 1A 0 F3 A7 S 1E BD % P 4 0F ) A0 D0 B ' 83 A 07 18 E7 CD K 81 A0 y % 94 FA l C6 D7 1C 0F 83 8 B7 e M 80 C0 0 8F 94 04 10 C 06 8B FA U , 18 06 A F0 1F 00 E9 F2 i w B 6 19 > N V R V R 19 F3 11 * BE y AC 03 01 80 r W E5 ~ 9B C2 09 ( F9 O C BD BA 1 F2 B0 09 # _ 06 00 P 03 A F A1 E4 FB 1D 10 i E0 DB B2 U   ` 1A 81 04 0F @ P * BD 2 x 10 D B1 . @ ` A 8D B9 AB 00 E2 FF FD I 7F F2 BE 1F : } E8 93 Q BB o _   E1 E0 ` C6 04 < AE 88 FF 08 9B K C1 80 16 / F0 8C 1 83 { 92 F J   ~ $ 17 8F 81 A1 FD 91 87 { 9C | t | l ; D7 89 0C 9D A0 4 0 Q 01 93 BC 06 86 0A 11 E6 o AF CF AF CF 8F F3 } } } } { 7F 9B EB E3 EB E3 E7 FA w B5 A0 ` 03 84 ` CD E9 x 18 00 E0 h 9F ^ 8C ? D4 E9 C7 G a 91 C3 FB D4 CE N CE N 9F 1E B6 t T 19 1C B8 A4 2 | } A9 x 05 9D 97 83 @ F8 t F4 FF CB C6 80 ( C8 1A 07 C2 C0 0A 3 06 81 F1 E1 FA E7 C6 n = 2 s CB FB r D3 p CC B4 D1 B7 BD 9 F9 E9 I C4 80 C8 L p BA 1C 13 { { FC C4 84 84 84 9F FF EC \ E2 C7 97 B8 B6 11 E7 & 86 A6 AF 6 t + BF * q p f P E3 00 0B 85 w C1 CB CD C3 # C3 C7 C ; J f p f n F8 g 8E j E2 D7 1D 8 B1 D0 EE FB ( 4 0C CA 1E j F8 7 8B 9C 1E < E5 00 3 85 v C1 FC y 8F 9E 8E { C6 E7 C5 : E8 BD E7 F 87 CE _ 0E EB CB FC F3 5 n 9 F0 f b DB A1 DD E1   1F 9F C F9 B4 X A7 U 9C F8 3 - 81 9A P 04 A1 x 02 E2 CF BA AA % 8F " 4 F0 05 95 1F 86 81 9B C6 H C8 C9 8F 92 1F 82 z A1 9 E2 B0 08 % % 83 { A1 p 4 10 f @ D0 B 13 13 C2 < D0 D4 1A 08 S 18 01 E 05 0 8F + 2 08 % C0 |   0F FC 98 a ) G _

Event Type:    Error
Event Source:    FMS (Core)
Event Category:    (263)
Event ID:    1029
Date:        8/26/2010
Time:        2:13:24 PM
User:        N/A
Computer:    XYZ
Description:
Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 81.242.158.174, App: LiveVideoStream3/400011064, Protocol: rtmp, Client: 5779300680901943606, Handle: 4258073711) : 85 ~ Q z 05 5 CB A0 } W 07 ~ 06 . 0A a T % 94 08 % DE T % 8F 81 80 q 04 ? 8F 81 80 87 9F 83 FE _ C5 c EF F1 E CB CC B7 w 0A 0F @ 8F 8B C0 FC F1 z AF EF BF % F5 M Z 7F FB C7 Z 06 16 09 9 V B7 19 m n 13 AB F3 03 E 96 , 19 BA 17 8F F5 B A8 B7 D 9 W 0E 1C C0 % h 1E U B8 C8 CC u D5 9B B6 0C E1 1D 8A I 03 A 07 1E 0D 04 1C 4 + E3 F3 DF 1A 0D 04 $ 95 1F 06 1F F1 CF F8 90 b F2 ~ 11 E7 99 A0 D0 03 P FC 1A 00 88 @ 08 86 06 82 11 DA ) EB AE V 0D 00 E E7 C1 A0 09 90 A0 D0 02 L FC | 3 t ? 1C | D5 ` D0 04 ED D0 h 01 i 95 83 @ 0F q F0 h 02 1C A3 B5 EF E0 11 9F f 0C 00 87 D4 C0 08 CD 12 C1 A0 07 B7 06 00 O CA ? 0F B9 S Q B4 1A 17 D7 CA CC AA 08 J k G 1F y & F 9D 05 B0 [ 08 q F2 D3 F0 % C5 C0 D0 A E9 FB A B5 81 7F 84 R 5 FF D0 CD FD EB BF E4 CB > D5 99 Z i 93 p b W BC > 1F * 12 84 89 8A 95 l 8C F2 95 94 $ 8 F7 E2 EF D0 h 7F 9C 14 { D3 y 13 87 E1 D 0B CF | F7 8E AE AA j y 1C k X EA L EE CE 0D \ CB BC = V D4 * ) 05 X C1 { A 91 81 AB 99 83 x / FE FE 12 FF FF 87 80 = F5 F9 E5 CC _ \ F8 FA 0D BF FB F3 DC 1B CC > 82 _ F9 D7 FF F0 W $ 1C 12 BC C4 > 90 ` C8 00 9E 87 = 05 06 EF 19 E3 95 80 T D5 06 80 12 1A 83 o FF 12 A0 CB B4 s EA CB C1 A0 07 9C 06 00 8 H C1 F9 r w C7 P h 01   81 80 0E F2 90 83 E5 BD 4 1B W F7 k N F B9 EC ` 84 A8 A1 l L } AD Z A1 ! S j 0A FA Z X 94 89 C8 18 06 F0 ` 1C ` @ F3 g n C2 08 0 0D ? FF C4 w BE CC E G E0 @ % EE 1E 06 02 1F FC F6 03 19 04 . i 04 ( R 14 E0 84   ` 1B 15 C2 F8 u BC 03 04 98 $ 8F BD F0 FF 0A P 2 F7 CF AA 00 87 F8 19 F6 \ < DE AF B7 88 97 J q 9F FF C5 83 C4 ( A6 13 B6 C3 p O ] A0 } E5 1E - L T 89 . 9 8D W EB 1A # FF 13 D B5 00 4 EF CC DF B9 EC 80 CA @ 90 B4 89 M E7 01 1C 6 e D0 8E C6 E0 BF C1 BC FF ~ C4 A0 B6 } < FE F6 F3 ? CC F0 / D8 0B 7 F4 # 93 EF 9F EF CC 14 $ 09 E8 1C 05 AB D4 , 90 B0 D0 X G 0A F1 13 A8 p \ C3 C ` , E2 A B6 FF w 8D | ] { 88 0 02 0A 80 F2 A5 b _ DB BA 05 81 86 02 B3 18 a c 0 c E1 * AB A6 DE @ < I F8 06 00 a 7F y 15 7F DE EA F1 85 02 w AA M 10 AB C6 Z E0 83 00 CC 0C 03 8F C4 90 82 AF DE j ? 9F 88 BE A9 W FA BD 7F ; 9D E3 A0 9C DF 9E t 18 02 00 0D CC R A8 10 AD 10 C5 | F9 @ C0 0B 09   84 0D 03 D3 B3 a 97 FD 7 DA , 0F E7 01 90 01 87 Y A2 A4 80 8B 0A 0 \ D6 % 17 82 1A A9 FE DA 0C 84 0D AB AC E6 , 06 B 08 E4 1B C9 DA F m s 8B 96 83 $ 02 e 8C : 0F 91 x h D1 FE ` E4 04 C3 : : CE B8 94 B0 A6 15 1D t C0 d 00 ^ E2 E7 18 ' 82 8E D8 95 0C 08 DE h 2 10 2 E9 03 A3 D3 F3 EB F0 F9 A5 84 AF E7   E1 D9 1A BD FF CC 9E 09 O 81 8F 81 | - / F1 < m ` 8E Q EF 0 o q 10 08 ~ \ 1B 7F F8 % J ( A6 f , 1A 00 W 97 C2 c 96 DF EF AA 12 95 [ AE AF 06 01 94 18 06 D0 ? A5 DE A6 DF 84 90 ` a H 4 C1 80 p 06 87 E9 DD S BF D1 F3 s s s ; BD F5 BF 6 F0 91 F1 E1 7F 9F CD C6 A8 G s 90 e FE F7 C1 83 00 . % 8F K A7 AA ` X B1 D 8B s J B1 r CE c E4 01 80 19 h 2 ` & 81 / CC m EF * 8D 1D 84 8 18 3 AB BE C1 1D B4 ( C4 1

Event Type:    Error
Event Source:    FMS (Core)
Event Category:    (263)
Event ID:    1029
Date:        8/26/2010
Time:        1:53:23 PM
User:        N/A
Computer:    XYZ
Description:
Bad network data; terminating connection : (Adaptor: _defaultRoot_, VHost: _defaultVHost_, IP: 77.84.92.167, App: LiveVideoStream3/400011064, Protocol: rtmp, Client: 5510270990406217012, Handle: 4127395405) : B8 8A ? y U Y DD   0F T 07 84 91 FA A0 0 19 B4 18 8A 00 A0 CF s k s e 9A 0C 3 D7 D5 D7 D5 f 83 0C F4 F4 t B4 19 A0 C3 < FC EC FC E6 X 0 CB 99 93 97 8C y 8E 14 9 C7 C4 C0 80 15 p 0D 04 # 0C 0 s 87 81 d 07 83 * E0 B 0C 98 a 83 9B FB DA C0 ` ! DD V 0C 04 A + 0C 8 s w s c D AA C1 DB l < C1 0E 19 DC [ [ DB 0 D 06 v B6 v B6 l 11 A 9D 95 8D 95 8A FC P g a _ _ \ C1 16 19 D7 V C6 D6 D6 B0 E 86 u 95 u 95 k F1 a 9D U M U J FC ` g Q O Q N BF 18 19 D3 D2 D3 R AF C6 86 t 94 t 94 8B F1 C1 9D % 15 1D 12 FC o 85 ^ ~ CF M x 0 01 A0 1F F6 83 , 7 9E D7 81 EE DE D2 FF 9F Y 1F 06 M C2 C 94 80 ) 94 1A 07 C9 FF 8 06 $ g " # A3 9 > T 0E 92 U DE = B1 1 Y 3 18 FF D8 19 B4 83 @ 09 R 01 A0 D0 C1 01 ! AD CC C E1 1D ) C6 96 a E8 ! 0E E9 86 + EF 1B 03 EF 81 80 12 12 AA B0 83 EF E0 E8 | A6 D4 AD H Z } l ! 03 C 05 10 % 17 y W 15 L 9B ~ 8D EB 82 A A6   0B 98 1E 80 K ; C3 0 A3 CE < 3 { 94 C3 0 0B 0C D0 P O D0 ] $ C 4 14 14 14 07 9 00 CD 01 00 F8 F9 C6 @ 0D 91 BF s B5 A9 DA 92 1B DE 12 D BF 84 1 , BC K W DF 97 AB BF E8 p i 80 18 06 81 & 04   ` 04 A 05 ^ E5 A2 R B1 F7 81 C6 83 FD 1F C1 E3 B9 9C 83 8 p h 01 00 DF 9A X 18 02 D0 ` 9C C1 84 07 06 99 F3 96 D 18 02 D8 0A   0D A0 C1 BE 0F A2 08 01 m EE 83 F9 A0 C8 90 9C 94 F5 06 E9 10 = Q F8 A0 C0 6 03 03 E0 10 A 83 FC 06 18 09 CA 8F D1 08 * C0 F8 C 12 D5 81 92 FF 8C Z 1D 18 01 A 9E DE CE D6 C3 , 18 g AF A9 AB A6 CB 08 03 9D - 0C F3 D9 . d = 08 1A f 1E C8 00 8C + E6 18 P E3 06 06 W g AE l < 19 / C3 07 DA C7 V C5 O A2 85 CE 99 1E 9 86 18 > 98 10 9F D 08 O 95 = ( 01 , P E1 F3 C0 84 19 CE A8 $ 8F 1E BA < B8 18 17 BF 8F 0B 80 D9 + 1C @ 1F ' EE 03 D 05 % 83 E5 ` D1 01 BD 09 00 C0 08 Z 0C 0F D5 R AE AD l F3 9D 01 80 89 12 D 82 F0 ` c BE AC H C0 ` 81 0B CB A9 ~ AF D2 F8 q 98 02 b B5 C0 : A5 12 B 10 ! v 08 D4 e f EE 10 C1 A = ` C6 0 10 E1 06 EF 2 0F 9A - 18 U 17 C9 / = FF @ T 9B b F3 E2 83 ; + 0B 0A E5 F8 A0 CE C2 BA B6 B1 ~ ( 3 AD AC AB AB ` 8A 0C EB * EA AA W E2 C3 : AA AA 9A 95 F8 C0 CE A6 A2 A2 A1 ~ 0 3 A8 A7 A8 A6 ^ 8D 0C E9 i A9 A9 W E3 C : Z J Z E E8 D0 CE 8E 8A 8E 89 ~ 6 / = 81 EA EF { / FA 90 h 1E D6 87 F7 15 ? FD W C 6 " E2 z C2 E7 01 EC A0 D0 02 A7 86 L 00 D0 02 A6 9 AB E6 E6 DC 19 A4 $ $ # B8 CB 01 E9 00 D1 ? * 04 1F 19 H 0C 91 84 BF h d D9 0E S 0F C7 g C0 F6 S CF 01 0F FB G 91 83 m C F5 ` z 12 FF C1 DB D8 < O d > . d 98 0F y 1F 00 R C0 4 00 8C E5 C1 92 90 d CC AD F3 8E 0C D2 D2 R Q 9C e 03 4 d D D W ) 4 F0 0B 0C DE 19 86 g E7 E7 A7 AE 92 01 88 F8 F1 E1 E1 DB 9C x f x ~ v | E3 1C 07 O C0 C6 C0 * E8 0 12 * 93 82 08 FB A 84 85 12 E FC AD E0 C0 / 09 @ A1 06 00 H K 06 0C 16 F8 I 06 12 0F t B0 3 E0 < D9 < ~ 1C 02 80 EF 8E 4 18 02 B0 ` b 02 08 0 7F ~ 06 16 09 FB A7 A6 01 80 ( W 90 03 U 01 q F7 CB > E6 z E8 j h 9 L S Q > AC F6 C9 0 : ` 00 F2 F0 P 04   86 ? 06 0A 10 J / V

I have lot of other example if you want

Thanks for your help, It's very appreciate.

Asa_-_FMS

Adobe Employee

Are you using a FlashPlayer to produce these Bad Network Data messages. Surprisingly there's little to any useful information about the connection when this happens, that usually points to a bad protocol stack on the far end connection.

Asa

A

Anonymous

We use Flash6 for the broadcaster (old legacy), and we have 2
Applications for the viewer, one in flash8 the other in Flex4

Are there a chance that the problem come from there?

A

Anonymous

Does someone got an answer to the problem mentioned
above?

- Asynchronous I/O operation failed

- Unexpected virtual message from client

I have the same problems in the event log of the server and
Most of the time clients are disconnected rather long
after with this error :

- Bad network data; terminating connection

I use FMS 3.5.4 on a Win2k3 server

Thanks a lot

Asa_-_FMS

Adobe Employee

Hi Franky,

Can you reproduce the errors below? Would you be willing to post some of the content of the BadNetworkData errors so we can determine the kind of network data FMS is encountering and thinking is bad?

Asa

D

dejanp777

Inspiring

I think problem started after I update FMS to latest 3.0.5, did same thing started happening to you after you upgraded to 3.5.3 ?

Asa_-_FMS

Adobe Employee

Hi to Amnl and dejan,

I'd like to take a support case for this and investigate what's going on here. Would either/both of you mind contacting me offlist at awhilloc@adobe.com. We'll get you a complimentary support incident and do some investigation to figure out the problem.

Asa

A

amnAuthor

Known Participant

Gladly!

Taking this by email from now on then ...

Wishing best of luck to anybody stumbling upon this thread, hoping that the issue will be resolved!

D

dejanp777

Inspiring

Same thing started happening on my Flash Media Interactive Server 3.0.5. version. Without any particular reason streams in my video chat application stop working, while other things on FMS work, at same time connections to FMS start piling up without going down as I can see in FMS admin console.
After about an hour FMS restarts automatically, after it's automatic or my manual restart everything works fine again.
I have checked all FMS logs and there is nothing unusual when this problem occurs, only thing that I noticed is that restart of FMS lasts much longer when this problem is present, and in master log I can see:
2010-04-16 04:42:57 1512 (e)2581327 Master shutdown has timedout, killing child process 3484 -
Does this mean that FMS Core become unresponsive? and do you have any idea how can this be solved or avoided?

A

Anonymous

I had a similar problem a while ago on a specific server... turned out that the antivirus software on the server was causing the problem in my case.

There's a technote about possible condition here:

http://www.adobe.com/go/kb402920

A

amnAuthor

Known Participant

Well, like I said, there is no antivirus - there are only Microsoft processes normal to a clean Windows Server 2008 install and the

FMS running.

A

Anonymous

right... but the technote I linked to explains what conditon causes the error... does not relate specifically to antivirus (that's just what it was in my case)

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.

Scanning file for viruses.

This file cannot be downloaded