• Global community
    • Language:
      • Deutsch
      • English
      • Español
      • Français
      • Português
  • 日本語コミュニティ
    Dedicated community for Japanese speakers
  • 한국 커뮤니티
    Dedicated community for Korean speakers
Exit
1

Meta Co-Tracker AI integration in After Effects

Community Beginner ,
Feb 07, 2024 Feb 07, 2024

Copy link to clipboard

Copied

Context:
I am trying to integrate Meta Co-Tracker (as of now, maybe will look for better ones too) in After Effects to track 2D points in one click with a single go, will explore more, or if you've suggestions let me know.
Currently, I managed to get the tracked points data using .json to AE shapes (Please see preview video).

Problem/Question 1:
I think creating this entirely AE plugin/script not gonna work due to performance reasons (or my cognitive limits), so can I create some GUI python program and link that to AE via SDK ... like Mocha, or Element 3D as they do their processes standalone and then export desired that data to AE which is efficient.

Problem/Question 2:
There are many plugins in AEScripts.com where they are using AI within AE plugin only, how they are doing it as a user don't have to install additional libs, python, etc? or is it doing some python process at the backend using SDK?

Current Progress Preview: https://vimeo.com/910721332/4b71632738

TOPICS
Experiment , How to , Scripting , SDK

Views

181

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
LEGEND ,
Feb 07, 2024 Feb 07, 2024

Copy link to clipboard

Copied

All AE cares about is whether it receives the data in a format it understands either via the APIs or using other methods like JSON or copy & paste. The rest is entirely up to you. Most of the tools you mentiuon simply set up a virtual memory host and then convert the data structures dynamically. Whether or how that will work with the Meta tracker is of course an open question, but I can't see why registering a bunch of property streams and shuffling around the image buffers shouldn't work. The same relates to the question of whether the code can be included in the plug-in itself or must be accessed in other ways. I'm sure the terms of use for the tracker code have something to say about this.

 

Mylenium

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Engaged ,
Feb 07, 2024 Feb 07, 2024

Copy link to clipboard

Copied

As to your second question, AI plugins on aescripts would probably be referencing a pre-trained model and adding that as a dependancy in their project. I don't have any experience doing this, but I am assuming it is similar to linking to a 3rd party library statically compiled so that the end-user doesn't need to install any extra dependencies themselves. When the final binary is built the IDE will compile it into one file/bundle.

 

You'll notice that the AI plugins are usually larger (potentially gigabytes each) than the average plugin (a few megabytes).

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Feb 11, 2024 Feb 11, 2024

Copy link to clipboard

Copied

You're right, I  tried to reverse engineer one of the Blace AI plugin and it has PyTorch, OpenCV 3rd party library linked to the ae plugin itself, so I now understand what it will take to develop such a plugin.

Chris Vranos demotivated me 😞  in your interview with him on your YT where he said how tedious/difficult the plugin dev in C++ for AE is, but still, I will see how much difficult it is for myself xD.


Anyways thanks James, your brief plugin tuts gave me a solid base to start developing and NT Productions (YT) is great for further deep learning. 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Expert ,
Feb 11, 2024 Feb 11, 2024

Copy link to clipboard

Copied

there's a new python api for AE, courtesy of @Trenton5EC4 !

it can shorten your development process for intergating your tech.

https://community.adobe.com/t5/after-effects-discussions/after-effects-python-api/td-p/14386368

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Feb 11, 2024 Feb 11, 2024

Copy link to clipboard

Copied

That is just CEP based plugin api, and doesn't include to manipulate render buffer things on AE, making it useless for my task.

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Explorer ,
Feb 11, 2024 Feb 11, 2024

Copy link to clipboard

Copied

That is 100% false. PyShiftAE has absolutely nothing to do with CEP. 

if you want to manipulate pixels in AE, you need to write an effect plugin. I haven't added that functionality to my plugin yet. 

you can, however, integrate your own python libs, use python GUI, and manipulate based on replacing source. But again, my plugin is still a work in progress by any means. 

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines
Community Beginner ,
Feb 11, 2024 Feb 11, 2024

Copy link to clipboard

Copied

LATEST

Yeah that's my point that it's right now not capable of manipulating pixel data yet and provides functionality similar to extendscript or a little bit more due to being made on C++ SDK.

 

So effect plugin is the only way as of now.

 

I hope more people can come to contribute to PyShiftAE to make its development faster, the whole community wants full python enabled AE SDK very bad 💀

 

I really appreciate the efforts you've and are putting in PyShiftAE

Votes

Translate

Translate

Report

Report
Community guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
community guidelines