FMIS 3.5.7: Segmentation fault in FcPlayInfo::cacheStreamMediaContexts(FcPlayInfo const&)

Hello,

we're running some FMIS 3.5.7 servers on which we are experiencing regular segmentation faults on high load. The stack trace from all of the crashes are identical:

Program terminated with signal 11, Segmentation fault.

(gdb) where

#0  0x082158ed in FcPlayInfo::cacheStreamMediaContexts(FcPlayInfo const&) ()

#1  0x082281b4 in FcRecordedStream::executeTransition(FcPlayInfo&, INetStreamPtr const&) ()

#2  0x0822e4c7 in FcRecordedStream::executeBufferCmd(FcPlayInfo&, INetStreamPtr const&) ()

#3  0x0822e6be in FcRecordedStream::execute(int, FcPlayInfo&, INetStreamPtr const&, int, bool) ()

#4  0x082155c2 in FcPlayInfo::execute(int, INetStreamPtr const&, int, bool) ()

#5  0x081ffba0 in FcNetStream::doBuffering() ()

#6  0x0830ca04 in TCClient::onQueueEvent(TCObjectPtr&, void*) ()

#7  0x081aee9f in QueueAPC::run() ()

#8  0x0844420c in TCThreadPool::launchThreadRun(void*) ()

#9  0x081be305 in FMSFiberPool::run() ()

#10 0x081be351 in FMSFiberPool::launchPoolRun(void*) ()

#11 0x081bce20 in FMSFiber::fiberStart(void*) ()

#12 0x00bb3b7b in __vstrfmon_l () from /lib/libc.so.6

#13 0x00ccfff4 in ?? () from /lib/libc.so.6

#14 0xe9f00010 in ?? ()

#15 0x0a14cba0 in ?? ()

#16 0xf5252198 in ?? ()

#17 0x00be6fb7 in _int_memalign () from /lib/libc.so.6

We're only serving VOD over RTMPE, almost all with Dynamic Streaming. One single app with no specific code. One authorization module which only authorizes TRANSFORM calls. No odd log entrys around the time of the crash.

This weekend we had 6 instances of the crash above. For every crash, our customers naturally experiences a connection drop, and although the player is able to automatically reconnect to our other servers, there is still an interrupt. The traffic spikes on the other servers are also undesireable.

Among the core dumps I've been looking at, I found another one which doesn't seem to be related. It seems to be connected to processing an incomming RTMP message (DOS attack vector?);

Program terminated with signal 11, Segmentation fault.

#0  0x0084bf5f in _int_free () from /lib/libc.so.6

(gdb)

(gdb) where

#0  0x0084bf5f in _int_free () from /lib/libc.so.6

#1  0x0084dfb7 in _int_memalign () from /lib/libc.so.6

#2  0xf7dad517 in ?? ()

#3  0x00000064 in ?? ()

#4  0xf11210d8 in ?? ()

#5  0x008493ad in _IO_str_seekoff_internal () from /lib/libc.so.6

#6  0xf7dad64d in ?? ()

#7  0x00000064 in ?? ()

#8  0x08754888 in asctime_r ()

#9  0xf1256ee8 in ?? ()

#10 0x084335cb in char* getString<char, TCStringPoolCtx, std::map<char const*, unsigned int, CharPoolCompare<char>, std::allocator<std::pair<char const* const, unsigned int> > > >(char const*, int, TCStringPoolCtx*, std::map<char const*, unsigned int, CharPoolCompare<char>, std::allocator<std::pair<char const* const, unsigned int> > >**) ()

#11 0x084335b7 in char* getString<char, TCStringPoolCtx, std::map<char const*, unsigned int, CharPoolCompare<char>, std::allocator<std::pair<char const* const, unsigned int> > > >(char const*, int, TCStringPoolCtx*, std::map<char const*, unsigned int, CharPoolCompare<char>, std::allocator<std::pair<char const* const, unsigned int> > >**) ()

#12 0x08432022 in TCStringPool::get(char const*, int) ()

#13 0x0842d05d in TCMBString::TCMBString(char const*) ()

#14 0x0842e1ae in TCString::toMultiByte() const ()

#15 0x083c8c3c in TCTinCanProtocol::processConnect(TCClientPtr&, TCMessagePtr const&) ()

#16 0x083cc208 in TCTinCanProtocol::processMessage(TCClientPtr&, TCMessagePtr const&) ()

#17 0x083cc32a in TCTinCanProtocol::onMessage(TCClientPtr&, TCMessagePtr const&) ()

#18 0x08415ea7 in TCSession::processRtmp(TCTinCanProtocol*, TCClientPtr const&, TCBuffer const&) ()

#19 0x0840d298 in TCRtmpSession::processRtmpData(TCBuffer const&) ()

#20 0x0840e2b8 in TCRtmpTransport::onReadComplete(TCBuffer const&, unsigned long) ()

#21 0x0844d46c in TCTinCanProtocol::onReadComplete(TCObjectPtr const&, TCBuffer const&, unsigned long) ()

#22 0x082cf552 in TCAsyncIO::addConnection(TCConnection*, TCObjectPtr&, TCBuffer const&, TCMessagePtr const&) ()

#23 0x083f5795 in TCProtocolAdaptor::import(TCBuffer&, TCBuffer const&, TCObjectPtr&, TCMessagePtr const&) ()

#24 0x083f8ffd in TCProtocolAdaptor::doMigrate(int, TCBuffer&) ()

#25 0x083faa1d in MigrationHandler::run() ()

#26 0x0844420c in TCThreadPool::launchThreadRun(void*) ()

#27 0x081be305 in FMSFiberPool::run() ()

#28 0x081be351 in FMSFiberPool::launchPoolRun(void*) ()

#29 0x081bce20 in FMSFiber::fiberStart(void*) ()

#30 0x0081ab7b in __vstrfmon_l () from /lib/libc.so.6

Adobe, any ideas?

Thank you

Johan