Integrate native WebRTC audio/video on all AIR platforms including Adobe Media Server

[flashdictionary]

Hey everybody,

It is apparent that Adobe is very busy trying to keep up with improving and fixing video and audio bugs on all platforms. Thankfully, in the past month Adobe finally made a H264 video with Nellymoser audio stream work on Androids! That took a while to make that work. But even though that is working, AIR on Androids still cannot transmit H264 video. Also AIR on iOS cannot view live video, but instead video has to be wrapped inside a Apple HLS (HTTP Live Streaming) format which introduces way too much latency for live audio/video streaming. Also AIR cannot transmit AAC Audio, and echo cancellation with Nellymoser just doesnt make the cut.

Everybody is aware that Flash Player can only stream video/audio smoothly for 1 in 10 users. There is just way too much for Adobe to do to get audio/video to work again and to work for everybody on every device.

So because WebRTC has much more development effort going into it, and because it is being promoted as free source to try to make the proprietary licensing world to rethink H264 and AAC audio, perhaps Adobe should just focus their efforts on implementing the existing and mostly working WebRTC libraries into Adobe AIR.

Since these libraries are becoming so popular, Adobe could also integrate support for WebRTC into the Adobe Media Server for recording, peer-to-peer negotiation, and firewall hole punching.

See my feature request here to integrate native WebRTC audio/video on all platforms

https://bugbase.adobe.com/index.cfm?event=bug&id=3728399

So will you vote with me to get WebRTC into Adobe AIR and Adobe Media Server?

Adobe could essentially deprecate many existing features requests and bug fixes related to audio video and solve many problems with WebRTC such as:

Implement Opus Codec:

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3016518

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3331640

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3633142

Fix Enhanced Microphone issues:

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3711062

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3521224

Add support to stream AAC HE-AAC v2 audio, allowing iOS to be an endpoint that can receive audio and video:

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3694393

Add support to transmit H264 video from an Android:

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3648262

Decreased video latency and more performance on Androids regardless of the 32-bit/64-bit ARM Intel processors

https://bugbase.adobe.com/index.cfm?event=selectBug&CFGRIDKEY=3648268

Improve video quality and bandwidth:

HEVC, H265, VP9

[nebbb]

As there been any update from Adobe regarding support for WebRTC?

[flashdictionary · Accepted solution]

No there has been no update nor comment from Adobe regarding adding support for WebRTC. According to this year's roadmap they are booked! So it appears that we will not be seeing WebRTC in the ActionScript platform this year unfortunately. Back in March I made some predictions as to what might be considered higher priority for Adobe to be working on. It appears that list is still being worked on by the Adobe AIR/Flash team. They still have not finished support for iOS 8, and they still have several months to get hardware accelerated video finished. And VideoTexture is still in beta and will probably be in beta until AIR version 19 I would guess. Then 64-bit AIR runtime will be completed 3rd quarter along with HTML5 improvments for the 4th quarter. It looks like Adobe is completely booked.

[flashdictionary]

I mentioned this to some colleagues and they had a hard time wrapping their minds around how this could be done, assuming that WebRTC is an HTML5 thing only. So allow me to clarify to everybody, that WebRTC is not an HTML5 thing. WebRTC is some low-level C/C++ Code that has been developed with compatibility in mind for Windows, Macs, iOS, and Androids. Many browsers are taking an initiative to integrate the WebRTC C/C++ code into their browsers and add API hooks so that people can use JavaScript to activate the native WebRTC code.

I am proposing that Adobe also takes the native C/C++ code and integrates into the Adobe AIR platform for Windows, Macs, iOS, and Androids. (There would no HTML5 involved with this, just low-level native code that performs outstandingly well.)

[flashdictionary]

Once the VideoTexture feature is fully integrated on Windows, Mac, iOS, and Android...(hooray Adobe, and great work)...the flash.display.Video object can become deprecated as well as the StateVideo object. How great is it now that we can use one a hardware accelerated video path that can be integrated into a fully GPU accelerated app with Starling and other GPU based ActionScript frameworks?!

The VideoTexture object supports the following methods now:

attachCamera(theCamera:Camera):void

attachNetStream(netStream:NetStream):void

Unfortunately though the NetStream class is bulky and not fully cross-platform capable due to shortcuts taken on iOS, and legacy code taken from ActionScript 2.0. Now imagine with me a new method added to the VideoTexture object that could help get rid of the NetStream bugs and limitations and provide an alternative (royalty free) streaming solution such as:

attachRtcStream(rtcStream:RTCStream):void

or

attachWebMStream(webmStream:WebMStream):void

[ramez_zegenie]

Hey flashdictionary,

Great posts, thank you.

Personally, I am leaning towards ANE's. The less we throw on the Adobe AIR team, the more they can focus on core issues (e.g. faster performance, support for Windows 8 phone etc.)

Given Google released the WebRTC source codes, how hard is it to make ANE's? They would be just wrappers. And for desktop, Chrome browser already supports it (in fact, for Android, newer Chrome's also support it -- so really iOS is the challenge).

I am also interested in this, maybe we can set up a crowdfunding initiative?

[flashdictionary]

I was hoping to get some support in creating an ANE for this. The Adobe AIR engineering team is so busy that they cannot get VideoTexture to work yet on iOS and Android, not even for the traditional Sorenson codec. I think that the only solution for video on mobile platforms is to create our own ANEs.

[ramez_zegenie]

I was under the impression videotexture works for h264 / aac - native video. That's normal, as it uses the GPU. Anything else, you have to do it yourself and use the CPU, which is why we would develop the ANE for WebRTC (which .. would use the CPU, at lease partially as the codec used by WebRTC is unlikely to be a closed one). It's not the job of Adobe AIR to support every single application. Some things have to go native.

So, do we start a crowdfunding campaign?

[flashdictionary]

Unfortunately VideoTexture is still failing for many use cases. It only works with a certain subset of videos and is continually crashing for me on iOS and Android. Because of this, it seems the only option to get solid and cross-platform consistent audio/video transmission and playing is through an ANE (Adobe Native Extension). If crowdfunding is the way to go, then lets do it.

[JankoMarkoDWCS3]

It seems that an ANE is the only way to go at this point, let's hope somebody make one for WebRTC for android and iOS

[Robert Mc Dowell]

let's do it...