/t5/after-effects-discussions/meta-co-tracker-ai-integration-in-after-effects/td-p/14407258Feb 07, 2024
Feb 07, 2024
Copy link to clipboard
Copied
Context: I am trying to integrate Meta Co-Tracker (as of now, maybe will look for better ones too) in After Effects to track 2D points in one click with a single go, will explore more, or if you've suggestions let me know. Currently, I managed to get the tracked points data using .json to AE shapes (Please see preview video).
Problem/Question 1: I think creating this entirely AE plugin/script not gonna work due to performance reasons (or my cognitive limits), so can I create some GUI python program and link that to AE via SDK ... like Mocha, or Element 3D as they do their processes standalone and then export desired that data to AE which is efficient.
Problem/Question 2: There are many plugins in AEScripts.com where they are using AI within AE plugin only, how they are doing it as a user don't have to install additional libs, python, etc? or is it doing some python process at the backend using SDK?
/t5/after-effects-discussions/meta-co-tracker-ai-integration-in-after-effects/m-p/14407458#M243664Feb 07, 2024
Feb 07, 2024
Copy link to clipboard
Copied
All AE cares about is whether it receives the data in a format it understands either via the APIs or using other methods like JSON or copy & paste. The rest is entirely up to you. Most of the tools you mentiuon simply set up a virtual memory host and then convert the data structures dynamically. Whether or how that will work with the Meta tracker is of course an open question, but I can't see why registering a bunch of property streams and shuffling around the image buffers shouldn't work. The same relates to the question of whether the code can be included in the plug-in itself or must be accessed in other ways. I'm sure the terms of use for the tracker code have something to say about this.
/t5/after-effects-discussions/meta-co-tracker-ai-integration-in-after-effects/m-p/14409502#M243709Feb 07, 2024
Feb 07, 2024
Copy link to clipboard
Copied
As to your second question, AI plugins on aescripts would probably be referencing a pre-trained model and adding that as a dependancy in their project. I don't have any experience doing this, but I am assuming it is similar to linking to a 3rd party library statically compiled so that the end-user doesn't need to install any extra dependencies themselves. When the final binary is built the IDE will compile it into one file/bundle.
You'll notice that the AI plugins are usually larger (potentially gigabytes each) than the average plugin (a few megabytes).
You're right, I tried to reverse engineer one of the Blace AI plugin and it has PyTorch, OpenCV 3rd party library linked to the ae plugin itself, so I now understand what it will take to develop such a plugin.
Chris Vranos demotivated me 😞 in your interview with him on your YT where he said how tedious/difficult the plugin dev in C++ for AE is, but still, I will see how much difficult it is for myself xD.
Anyways thanks James, your brief plugin tuts gave me a solid base to start developing and NT Productions (YT) is great for further deep learning.
That is 100% false. PyShiftAE has absolutely nothing to do with CEP.
if you want to manipulate pixels in AE, you need to write an effect plugin. I haven't added that functionality to my plugin yet.
you can, however, integrate your own python libs, use python GUI, and manipulate based on replacing source. But again, my plugin is still a work in progress by any means.
Yeah that's my point that it's right now not capable of manipulating pixel data yet and provides functionality similar to extendscript or a little bit more due to being made on C++ SDK.
So effect plugin is the only way as of now.
I hope more people can come to contribute to PyShiftAE to make its development faster, the whole community wants full python enabled AE SDK very bad 💀
I really appreciate the efforts you've and are putting in PyShiftAE
7
Replies
AdChoices