Using StageVideo and Stage3D in AIR - BETA

[chris.campbell]

We're planning on introducing a new ActionScript feature that will allow hardware accelerated video to be used as a source texture in a Stage3D environment.  Currently, using video with Stage3D requires the use of the Video object, which is not accelerated, and manipulation of the bitmap representations of the video frames.  The planned feature, called VideoTexture, will allow direct access to a texture object that is sourced from a Netstream or Camera object.

The following sample code demonstrates the use of the VideoTexture object.

1) Create a VideoTexture object and attach a NetStream (or Camera) object to the VideoTexture object:

var ns:NetStream;
var context3D:Context3D;
var texture:VideoTexture;
texture = context3D.createVideoTexture();
texture.attachNetstream(ns);
ns.play("video.3gp");
texture.addEventListener(VideoTexture.RENDER_STATE, renderFrame);

2) A texture representation of the current video frame can be retrieved from the callback function for the VideoTexture.RENDER_STATES event.

function renderFrame(e:Event):void
{
    // Render on stage3D with VideoTexture
}

Please note that this will be an "extended beta" feature.  The initial implementation, available  in our AIR 15 beta, will be for Windows/AIR only.  We are committed to expanding this feature to AIR mobile and Mac in a following release.  We'll also consider an implementation in Flash Player if there is sufficient demand.

Getting your feedback will be critical to making sure we're providing a solution that you can use.  Please let us know what you think and keep an eye out for updated beta release notes in the next couple of weeks.

[itlancer]

Hi Shashank,

There is another bug if you try to create VideoTexture when application window is not active: Bug#3938986 - [Windows] Create VideoTexture when application window is not active cause video not re...

May be you already notice it in release notes: "Screen does not render properly when enter foreground from background. (Bug 3919996)"

Anyway, related issues:

Bug#3860955 - Application crash

Bug#3759420 - [Platform_Windows] GPU accelerated video stop playing when screen resolution changes a...

Bug#3938906 - [Windows] Play video by StageVideo in inactive window cause white screen but sound pla...

Thanks.

[saschaweissbach]

I've been trying to get support for this feature for many years now. Please integrate this into mac and mobile app.
Great News!

Best Regards,

Sascha Weißbach

[DMHV]

This looks good.  What are the limitations of the video texture?  Can I use it like a regular texture for objects?  Can AGAL shaders be used on VideoTextures?  This feature could open several possibilities if implemented correctly.

[datee74z]

context3D.createVideoTexture() always return null, and Context3D.supportsVideoTexture is always false.. On Win AIR release 15.0.0.302 (and in the beta)..

Many people with the same problem... why?

[datee74z]

Yes! The update today with now AIR 15.0.0.328 fixes this problem and it now works great

Thanks Chris and Dev Team

[V_Zhelezny]

Hi

I build demo with VideoTexture, its looks very perspectively.
But what about support alpha channel?
When I use FLV with alpha channel - all alpha "pixels" has black color

In this document says about YUV->RGB conversion.
Have any more infomation about VideoTexture and alpha channel?

[itlancer]

Hi.

After configuring Context3D buffer write this line:

context3D.setBlendFactors(Context3DBlendFactor.ONE, Context3DBlendFactor.ONE);

[V_Zhelezny]

I added code with setBlendFactors, but nothing happened.

private function contextCreatedHandler(event : Event) : void {
    Starling.context.setBlendFactors(Context3DBlendFactor.ONE, Context3DBlendFactor.ONE);
}

I use Starling, maybe problem in code which used Starling classes ConcreteTexture or Image

var context : Context3D = Starling.context;
var vTexture : VideoTexture = context.createVideoTexture();
vTexture.attachNetStream(vbController.netStream);
var cTexture : ConcreteTexture = new ConcreteTexture(vTexture, Context3DTextureFormat.BGRA, 1600, 878, false, true, true);
var image : Image = new Image(cTexture);
addChild(image);

Using Windows and AIR SDK beta 15.0.0.349

Thanks!

[Leafcutter]

I realise I'm a bit late on this but being able to mix video and stage3d (starling) on iOS (which is what I understand this might ultimately provide) would be great.  I have a lot of stage video based apps where I have struggled to create a smooth enough UI experience using standard AIR graphics.  Being able to mix a video stream with a starling UI would be perfect.

Can you just get it released for iOS now please?

[rewb0rn]

Not a lot of video experience here, but should you not be able to use StageVideo in the back with Starling UI on top? Or is StageVideo not yet supported for mobile?

[Leafcutter]

I did get a small starling window working on top of a stagevideo on mobile.  The problem (apart from it taking a very long time for me to work through which settings were needed and which callbacks had to be implemented) was that the starling content couldn't surround the video.  The starling layer fully obscures any stagevideo it overlaps.

So if the starling is full screen then the video is hidden.  If the starling is a clipped rectangle then it can sit on top of the video, but you couldn't have a starling title bar above, list next to and content to the right of the video window.  You could effectively only have a starling view in a rectangle sitting on top of the video, when in fact I needed a stageVideo rectangle sitting on top of the starling (if that makes sense).  I wanted to produce something like the following screenshot but with a starling UI.

If we can paint video onto a texture and then sit that in the starling display list that will be perfect.

[Colin Holgate]

Could you achieve that by having several Stage3D areas?

One easy way to get the final effect would be to use GPU render mode, and the regular Flash display list. That can out perform Starling, and you can have transparent areas through to the StageVideo.

[henke37__]

You need four regions to frame a single video, that means four contexts. Weak devices only offer one and the maximum is four. And the contexts can't share anything, especially not textures.

So while it might be technically possible, might, it is unfeasible in practice.

[Leafcutter]

I did think about having several stage3d areas but then wasn't sure about performance, especially on an older ipad.  I didn't investigate it too much as I found it quite tricky to get the stageVideo and starling working stably and consistently across all ipads/ versions of iOS.

If I put the standard flash in GPU mode then does stage video still work?  I seem to think I had to have renderMode as direct in order for it to work efficiently and be stable.  I haven't been able to get much in the regular flash display list to outperform starling for most of the things I've tried - it seems pretty efficient.

[Colin Holgate]

StageVideo works well in GPU mode. If you use the same limitations in the regular display list as you have to do in Starling (like, don't expect long unique frame animations to work, and have your bitmaps or vector symbols on stage before you start to animate them), then the performance should be as good as Stage3D. It all ends up being textures on the GPU, only with Starling you have to do more work to do typical Flash timeline animations.

Here's an app I did that is GPU mode, has StageVideo, and ordinary display list art, set to 60 FPS:

Let’s Get Ready! on the App Store on iTunes

[Leafcutter]

I'm not doing any traditional flash timeline animation, it is all bitmaps and bitmap data.  When I discovered Starling I thought my troubles were over as everything I wanted to do (like create user interface components like scrolling lists and bitmaps) was done and worked much more smoothly than I was able to get with standard flash.  However it didn't work with StageVideo so I just reverted to creating it all myself with bitmaps etc.  Performance is ok but nothing like native and not as good as starling.  I was about 1 project away from moving everything over to Swift but if this stageVideo/3d thing gets released soon for iOS then that might not be so necessary.

[LittlePenguin]

Chris, great job, it is indeed one of the most needed features that the Air scene needs right now.

Any updates on expanding it to mobile too?.

[chris.campbell]

Hi Lea,

Look for an iOS beta hopefully a little later this month!

Chris

[Leafcutter]

Chris - any update on an ETA for iOS?

[itlancer]

And what about VideoTexture for Android?

[chris.campbell]

iOS in a couple of weeks (before then end of January).  It should be available in our first v17 beta.

Android is up next and we know there's demand!  I don't have a date for when this will be out in beta but it's a top priority for the team.

[kaveri2]

Thank you for the pre-beta info! Now that you've figured out how to render h264 to texture, you should be able to render to Video element (renderMode=GPU) as well? Any comment on this?

Android already supports h264 Video element, but it's maybe not rendered through AIR? I've seen some strange bugs when transforming h264 Video element, which suggest that it's directly rendered by OS.

[LittlePenguin]

Hi Chris, is this feature ready for iOS now?, I'm currently using StageVideo for iOS (iPad 2) and it works just fine, no complaints.

The problem I'm having is on Android, even when using a high end tablet (Samsung Tab S 10.5 T800) I'm getting a black screen the first time I try to play a video using SategeVideo, the sound plays though. After I stop and click play again the video sometimes plays, others it doesn't, it seems pretty random.

So it would be nice to have a solution for both worlds using VideoTexture.

Thanks.

[Leafcutter]

I've been using StageVideo on iPad without problems for a while too - it was a bit difficult to get it working at the start across all ipads (some worked fine and others didn't) but eventually I got something that was pretty stable.  However, one of my apps is universal and so runs on iPhone too and I just recently noticed that running under iOS 8 on an iPhone 6 I have started to see the dreaded black screen / still hear sound problem.  It seems to happen at random and after a day or so trying to fix it I just couldn't eradicate it.  A robust video solution to replace stageVideo would be great.

(Have to admit that for the app in question I have started investigating how hard it would be to port it to native code in swift)

[LittlePenguin]

I didn't have any problems in my iPad 2 when using StageVideo, however now that I remember my tests were made when the iPad was using iOS7.

Now with iOS 8 installed I tested it again, and StageVideo works without any issues.

Can't comment on newer versions of iPad, like, 3, 4 Air 1 & 2, etc, but I want to assume it would work at least the same.

However on Android.. I will spend my entire Sunday with this problem, * sigh intensifies *.