Streaming software development for fms

HI All. I have a customer that is actually developing an streaming solution in RTMP. So far, while trying to stream H264 files to FMS, he has some issues.

He defines those issues as:

He uses two video streaming services from his encoder (the one he is developing), both using same source and same h264 encoder. One of the streams use flv muxer and one uses mp4 muxer (f4v).
He has a component for the RTMP streaming (using librtmp) in order to streaming to the Flash Media Server.

There is two main issues:
1. In case of using mp4 mux, we can not send to the FMS b-frames (removed by the encoder, using profile base), If we send b-frames the FMS
disconnects the connection.
2. The FMS losses huge amount of memory while streaming back mp4's.

Any idea of what kind of data or logs should I monitor? Can someone point me to any related spec in order to know what could go wrong here?

Thanks,

Chaim