OpenCV and AfterEffects

[kyleh6002942]

Hello,

I am looking for some guidance for integrating OpenCV with the AE SDK.

Notes:

1. I've read through most of the (excellent) documentation and successfully experimented with all of the sample projects.
2. I've successfully built AE SDK linked with OpenCV4.1 dll's and can import and instantiate the basic OpenCV building blocks (eg. cv::Mat).
3. I assume some of the native AE functionality uses OpenCV since I can see some opencv dll's in the AE binary folder. Also, plug-ins like 3D Camera Tracker likely use feature detection and tracking from OpenCV.

Where I'm stuck is copying input data to a cv::Mat object so that I can leverage the OpenCV library for computer vision.

After trying for a few days I'm quite stuck with memory allocation errors etc. and it is quite clear from the AE SDK docs that allocating your own memory without using the AE SDK macros is a recipe for disaster (fair enough).

I'm wondering if there is any hope for trying to accomplish the following (copied from the Render() function of the AE SDK Skeleton example):

static PF_Err 
Render (
PF_InData *in_data,
PF_OutData *out_data,
PF_ParamDef *params[],
PF_LayerDef *output )
{
PF_Err err = PF_Err_NONE;
AEGP_SuiteHandler suites(in_data->pica_basicP);
// get input frame pixels from in_data
PF_EffectWorld *inputP = &params[0]->u.ld;
// initialize 4-channel 8-bit Mat
cv::Mat mat = cv::Mat((int)output->height, (int)output->width, CV_8UC4);
// copy pixels to opencv mat 
err = copyInput(&mat, inputP);
// do some computer vision stuff
err = doAwesomeComputerVision(&mat);
// copy opencv mat to output
err = copyToOutput(&mat, output); 
return err;
}

Any guidance is greatly appreciated.

Thank you!

[Guest]

You are almost done then, you just have to implement the copyInput() and copyToOutput() functions in your examples .

No, but seriously, this shouldn't be too much work.

You are using the old Render() framework, in future you should switch to SmartRender to support other bit depths and get better optimisation/performance, but this will do for now for testing. When using Render(), you will get the pixel data as 8bpc uncompressed values in an interleaved way with the channel order being BGRA. There will also be some unneeded data (for you) at the end of each row, so be sure to take the rowbytes value in account which will give you the amount of actual bytes per row, so you know how much to skip over.

Apart from that, it is pretty simple and you can use a function like this to get a pixel:

inline PF_Pixel* getPixel(PF_EffectWorld* inputP, const A_long x, const A_long y)

{  return (PF_Pixel*)((char*)inputP->data + (y * inputP->rowbytes) + x * sizeof(PF_Pixel)); };

The simply copy the pixel values channel by channel to your matrix.

Setting a pixel in the output follows the same principle, read from your matrix, copy it to the output buffer.

Warning: this will be very slow if you do it in a linear pixel-by-pixel fashion, so again, just use this for testing. For actual implementation, you should then use a multithreaded pixel copying process, either by using your own thread functions or via the provided AE iterate suites. These will allow you to process a bunch of lines per thread. Check the examples in the SDK, e.g. SDK Noise for that (or the forum here).

[kyleh6002942]

Hi Toby,

Thank you for the prompt reply, much appreciated.

I will explore the SmartRender() framework and attempt a naive implementation then move to a threaded approach.

I'll be sure to post the code here for others to reference and mark your answer as correct once I am able to implement.

Thank you!

Kyle

[bfan1256]

Hey! Would love to see the code that you have made if you can post this 😄

We have a project that needs to do something similar. 

Cheers,

Brandon

[ShiveringCactus]

I agree, I'd love to you what you've come up with.  I don't have a project I need it for, just curious