Converting video to bitmapdata sequence woes

Report · Apr 11, 2014

I have been struggling with a problem for a few days now. I feel like I've exhausted every possible means of achieveing my goal.

This is going to be long. I apologize.

Here's what I want:

I have 50-80 FLV videos (the total for the end product is undecided) that are on an external server. I want my application (when requested) to download an FLV, go through it frame by frame and convert each frame into a bitmapdata.

The result I need is exactly this, it is set in stone and vital for my project.

My first method was as follows:

I used netstream.appendBytes with my FLV bytearray but the problem with this is that believe it or not, it is IMPOSSIBLE to step through an FLV frame by frame. The best I can do is draw the video object to a bitmapdata every frame at the enterFrame framerate. The problem with this is that for large videos (1920 x 1080 for example) it will almost certainly take longer than 1/30th of a second, so the next frame might be missed. In testing this I was unhappy with the results and also found that it used more ram than it needed (frames would often get recorded twice).

In summation, this method was very messy and not at all elegant, especially considering that all I need is a step function that actually works! (netstream has a step() function but ONLY for content on Aodbe Media Server for some bizarre reason)

My next method was this:

Using Flash, I tried embedding an FLV in the timeline and exported it as an SWF. This worked just fine. I can step through it precisely and at my own pace - perfect... but unforunately I have to then repeat that process 50-80 times with each FLV, and also note that these videos are likely to change every now and then when the product is finished.
I tried to automate the process with a jsfl, but - as if Adobe were purposely trying to make it harder on me - that was impossible. You'll notice you get a wizard style window when you import an FLV into Flash, which is not possible to control with jsfl.

So I went searching for another solution.

Another method:

I found that I could convert FLVs to SWF with FFMPEG, so then I wrote a batch file (later converted to python because of my unfamiliarity of windows batch code) which did this automatically for me and considerably quicker than the last method (around 10 seconds to do 20 FLVs).

Then I uploaded them to the server and gave it a try.

Despite having a fully working crossdomain that allowed everything else, I was getting a security error whenever I tried to access the Loader.content.

Then I remembered, swf communication needs Security.allowDomain()............. which is impossible to do with FFMPEG

I was really REALLY starting to lose my patience by this point.

THE NEXT F**KING METHOD:

I now had 50 swfs which couldn't do shit but spew errors at me. I figured all I want to do is give them a document class with Security.allowDomain("*"), but how exactly do I go about doing that?

So I wrote this class:

package

{

import flash.display.Sprite;

import flash.system.Security;

import flash.text.TextField;

[SWF(backgroundColor = "#FFFFFF", width = "100", height = "100", frameRate = "30")]

public class test extends Sprite

{

[Embed(source="EXAMPLE.swf")]

static private const DATA:Class;

public function test()

{

Security.allowDomain("*");

addChild(new DATA());

}

(EXAMPLE.swf is a converted FLV from FFMPEG.)

Right. So I compile this with mxmlc.exe like so:

mxmlc test.as -static-link-runtime-shared-libraries=true

I upload the outputted test.swf to my server.

I then wrote the following code :

...

Security.loadPolicyFile("url to server crossdomain.xml");

var context : LoaderContext = new LoaderContext();

context.allowCodeImport = true;

context.checkPolicyFile = true

var loader:Loader = new Loader ();

loader.load(new URLRequest("url to my swf"), context);

...

Followed by a listener to grab the content and all that stuff that goes without saying.

But guess what... It didn't work. Again.

I get the incredibly helpful error of: SecurityError: Error #2121: Security sandbox violation: BitmapData.draw: [local path to swf] cannot access [url to swf]/[[DYNAMIC]]/1. This may be worked around by calling Security.allowDomain.

Let me repeat that - it told me "This may be worked around by calling Security.allowDomain"

I ALREADY DID THAT YOU IDIOT!

Finally, I tried the same thing but this time embedded the SWF as a bytearray in the document class above, and then using a loader with loadBytes. However - Same. Bloody. Result.

And then I wrote this as I can't think of any other way of converting these videos into bitmapdatas accurately.

I really don't want to do the first method, it's too innacurate.

The second method is a load of arse, even though it works. It takes a few minutes for each FLV (setting the framerate, doc class) and is prone to me making a mistake.

The method needs to be done in batch. When the product is finished and SWFs need to be changed, it needs to be quick and easy as it probably wont be me updating them and I don't wnat the hastle anyway. A simple batch / executable does the trick.

Obviously this would be a piece of piss if netstream worked (why doesn't it work?), or if the security functions applied to embedded SWFs (why don't they?)

I'm not expecting much help as I already have 2 shitty solutions but neither of which I'm at all happy with.

Surely though there must be some way of converting an FLV to a perfect sequence of bitmapdatas. No missing frames, and will run the same on differently specced computers.

Is there any way of changing an SWF setting that will do the equivalent of Secuirt.allowDomain("*")?

Any help would be nice though. There has to be a solution in here somewhere.

...And I'm sorry to repeat myself but I just can't get over this point - why is netstream.step() such a deceitful piece of shit?

Report · Apr 11, 2014

Since it works fine for you as an SWF but not an FLV, why not try mass converting them all using something else, and reuploading the SWFs to the server, and then using flash to get the frames that way, and using arrays to not skip any frames?

Report · Apr 12, 2014

This is what I did, but I have yet to find a way of exporting the swf with a document class with Securit.allowDomain("*") using FFMPEG or anything similar. I'm not sure the flv to SWF conversion works like that anyway.

I was able to convert the frames to bitmapdata when testing in a trusted environment (in my flashdevelop project folder), but when I move the swf anywhere else it gives me errors.

Report · Apr 12, 2014

Do you host video containing swfs on a different domain relative to the loading swf? If so, does server where video swfs are hosted have crossdomain policy xml?

Report · Apr 13, 2014

The videos are hosted on a server

The swf that runs them is run locally, not on a server.

For downloading images from the server, I need a crossdomain and this works fine.

For downloading and manipulating SWFs I need the SWF to also say Security.allowDomain("*"), otherwise as soon as I try and Bitmapdata.draw() the SWF movieclip it gives me a secuirty error.

I've been thinking of possible solutions to this entire problem.

This is all I can think of:

1) Somehow convert a flv to swf with a document class.

2) Find the equivalent as3 code for embedding an FLV to the timeline

3) Use this Adobe Media Server thingy which I just don't understand the purpose of so I can step frame by frame through an FLV.

4) Literally decode the FLV bytearray into bitmapdatas... Probably doable, but I can't find anyone who's managed to convert anything other than the keyframes.

The way I understand it, there's a keyframe every few seconds usually in a video file. The keyframe is like a jpeg, a full image. Between frames any difference/movement in the video image is corrected with blocks. So if I have a black box moving left and right slowly, each frame is just a fraction of difference, applying new blocks to the left and right side of the moving box. This obviously means it's a fraction of the size, because it's not having to rewrite unchange blocks in the image.

Now surely there's a way of interpretting a byte array (much like video player software) frame by frame? It's probably confusing as hell, but it must be doable right?

Report · Apr 13, 2014

1. I don't think playing with byte arrays will help you. What is captured with BitmapData.draw() is an interpreted pixel data - array of pixels that is a result of decoded video data that is performed . I am not saying it is impossible - it is extremely difficult to say the least, especially because you will need to gain a thorough knowledge of video file format. It just may be cost efficient to place videos on timeline manually.

2. Over years many people (including me) unsuccessfully attempted frame-by-frame video synching/frame reading without embedding videos on the timeline. If frame precision is a mission critical aspect - it seems there is no way around video embedding.

3. As an extension of 2 - anything else (using NetStream) will never deliver desirable precision. Some of the reasons are stemming from how video is compressed (you got it almost right). Using step() will not help - there is always some level of frame approximation. Video formats do not have frame notion similar to Flash. In a way video codec is trying hard to get rid of frame notion as hard as possible and describe data in more abstract manner - thus, again, lack of the capacity for frame-by-frame playback in the compressed video file. It is sorta like fitting of decompressed video back to frames realm is delegated to the client.

4. Perhaps the only way to achieve pension without embedding is to encode video with each frame being an i-frame (key frame) which is impractical because video will be blown to a humongous size.

5. As you said - you explored JSFL options but, still, I have a feeling it is possible. I have nothing to say here though because I am not familiar with JSFL much.

Report · Apr 17, 2014

I'm happy to report that I found the solution to my problem a few days ago

I found a useful binary called hxswfml (essentially SWFMill but written in Haxe and primarily for AS3).

There's no documentation about this, but by looking at the examples I saw you could define an FLV as a sprite like so:

input.xml:

</swf>

compiledDocumentClass.swf consisted of a compiled .as containing:

[SWF(backgroundColor = "#FFFFFF", width = "100", height = "100", frameRate = "30")]

public class ImageRoot extends MovieClip

{

public function ImageRoot()

{

if (Security.sandboxType != Security.APPLICATION) {

Security.allowDomain("*");

Security.allowInsecureDomain("*");

}

Then you just run:

hxswfml xml2swf input.xml output.swf

All I had to do was change the path of the video in the xml between each compile.

Now all I have to do is double click a batch file and within a minute it's compiled 100 security error free swfs!

When I first tried this method it didn't work, but then I changed the useNetwork attribute to "true" and that did the trick! (it might be that my earlier methods which used embed tags might have worked if I made sure -useNetwork=true in the compiler arguments, but this method is faster so I didn't bother to check)

Needless to say I was elated when I finally got this shit working. I was just about ready to give up, so this came as a great relief.

I hope this helps someone with a similar problem. Videos in flash are just an absolute nightmare, but at least I've found a way to work with the FLV kind...