OpenCV and AfterEffects

New Here ,
Aug 13, 2019 Aug 13, 2019

Copy link to clipboard

Copied

Hello,

I am looking for some guidance for integrating OpenCV with the AE SDK.

Notes:

  1. I've read through most of the (excellent) documentation and successfully experimented with all of the sample projects.
  2. I've successfully built AE SDK linked with OpenCV4.1 dll's and can import and instantiate the basic OpenCV building blocks (eg. cv::Mat).
  3. I assume some of the native AE functionality uses OpenCV since I can see some opencv dll's in the AE binary folder. Also, plug-ins like 3D Camera Tracker likely use feature detection and tracking from OpenCV.

Where I'm stuck is copying input data to a cv::Mat object so that I can leverage the OpenCV library for computer vision.

After trying for a few days I'm quite stuck with memory allocation errors etc. and it is quite clear from the AE SDK docs that allocating your own memory without using the AE SDK macros is a recipe for disaster (fair enough).

I'm wondering if there is any hope for trying to accomplish the following (copied from the Render() function of the AE SDK Skeleton example):

static PF_Err

Render (

PF_InData *in_data,

PF_OutData *out_data,

PF_ParamDef *params[],

PF_LayerDef *output )

{

PF_Err err = PF_Err_NONE;

AEGP_SuiteHandler suites(in_data->pica_basicP);

// get input frame pixels from in_data

PF_EffectWorld *inputP = &params[0]->u.ld;

// initialize 4-channel 8-bit Mat

cv::Mat mat = cv::Mat((int)output->height, (int)output->width, CV_8UC4);

// copy pixels to opencv mat

err = copyInput(&mat, inputP);

// do some computer vision stuff

err = doAwesomeComputerVision(&mat);

// copy opencv mat to output

err = copyToOutput(&mat, output);

return err;

}

Any guidance is greatly appreciated.

Thank you!

TOPICS
SDK

Views

1.7K

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Guest
Aug 14, 2019 Aug 14, 2019

Copy link to clipboard

Copied

You are almost done then, you just have to implement the copyInput() and copyToOutput() functions in your examples .

No, but seriously, this shouldn't be too much work.

You are using the old Render() framework, in future you should switch to SmartRender to support other bit depths and get better optimisation/performance, but this will do for now for testing. When using Render(), you will get the pixel data as 8bpc uncompressed values in an interleaved way with the channel order being BGRA. There will also be some unneeded data (for you) at the end of each row, so be sure to take the rowbytes value in account which will give you the amount of actual bytes per row, so you know how much to skip over.


Apart from that, it is pretty simple and you can use a function like this to get a pixel:

inline PF_Pixel* getPixel(PF_EffectWorld* inputP, const A_long x, const A_long y)

{  return (PF_Pixel*)((char*)inputP->data + (y * inputP->rowbytes) + x * sizeof(PF_Pixel)); };

The simply copy the pixel values channel by channel to your matrix.

Setting a pixel in the output follows the same principle, read from your matrix, copy it to the output buffer.


Warning: this will be very slow if you do it in a linear pixel-by-pixel fashion, so again, just use this for testing. For actual implementation, you should then use a multithreaded pixel copying process, either by using your own thread functions or via the provided AE iterate suites. These will allow you to process a bunch of lines per thread. Check the examples in the SDK, e.g. SDK Noise for that (or the forum here).

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Aug 14, 2019 Aug 14, 2019

Copy link to clipboard

Copied

Hi Toby,

Thank you for the prompt reply, much appreciated.

I will explore the SmartRender() framework and attempt a naive implementation then move to a threaded approach.

I'll be sure to post the code here for others to reference and mark your answer as correct once I am able to implement.

Thank you!

Kyle

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
New Here ,
Sep 02, 2022 Sep 02, 2022

Copy link to clipboard

Copied

Hey! Would love to see the code that you have made if you can post this 😄

 

We have a project that needs to do something similar. 

Cheers,

Brandon

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Sep 04, 2022 Sep 04, 2022

Copy link to clipboard

Copied

LATEST

I agree, I'd love to you what you've come up with.  I don't have a project I need it for, just curious

Likes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines