Video playback lagging while buffering

Having a problem with video in my iPad app. The video is loaded from a server into an FLVPlayback component. While the video is buffering, the video playback is extremely choppy (like 1-2 FPS), but as soon as the buffering is complete, it plays smoothly. So obviously, the buffering itself uses too many resources (I've tested both on an iPad Mini and an ancient iPad 1).

Is there any way to reduce the resource usage of the buffering process so I can start playing before buffering is complete? YouTube is able to buffer while playing the same devices, so it should be possible. The videos are pretty low quality and around 70mb, and the bufferTime is default, not that that should make a difference.