Skip to main content
Izaan Ishaq
Izaan Ishaq
Participant
February 7, 2024
Question

Meta Co-Tracker AI integration in After Effects

  • February 7, 2024
  • 3 replies
  • 1233 views

Context:
I am trying to integrate Meta Co-Tracker (as of now, maybe will look for better ones too) in After Effects to track 2D points in one click with a single go, will explore more, or if you've suggestions let me know.
Currently, I managed to get the tracked points data using .json to AE shapes (Please see preview video).

Problem/Question 1:
I think creating this entirely AE plugin/script not gonna work due to performance reasons (or my cognitive limits), so can I create some GUI python program and link that to AE via SDK ... like Mocha, or Element 3D as they do their processes standalone and then export desired that data to AE which is efficient.

Problem/Question 2:
There are many plugins in AEScripts.com where they are using AI within AE plugin only, how they are doing it as a user don't have to install additional libs, python, etc? or is it doing some python process at the backend using SDK?

Current Progress Preview: https://vimeo.com/910721332/4b71632738

    3 replies

    S
    Community Expert
    shachar carmiCommunity Expert
    Community Expert
    February 11, 2024

    there's a new python api for AE, courtesy of @Trenton5EC4 !

    it can shorten your development process for intergating your tech.

    https://community.adobe.com/t5/after-effects-discussions/after-effects-python-api/td-p/14386368

    Izaan Ishaq
    Izaan IshaqAuthor
    Participant
    February 11, 2024

    That is just CEP based plugin api, and doesn't include to manipulate render buffer things on AE, making it useless for my task.

    T
    Trenton5EC4
    Inspiring
    February 11, 2024

    That is 100% false. PyShiftAE has absolutely nothing to do with CEP. 

    if you want to manipulate pixels in AE, you need to write an effect plugin. I haven't added that functionality to my plugin yet. 

    you can, however, integrate your own python libs, use python GUI, and manipulate based on replacing source. But again, my plugin is still a work in progress by any means. 

    James Whiffin
    James Whiffin
    Legend
    February 8, 2024

    As to your second question, AI plugins on aescripts would probably be referencing a pre-trained model and adding that as a dependancy in their project. I don't have any experience doing this, but I am assuming it is similar to linking to a 3rd party library statically compiled so that the end-user doesn't need to install any extra dependencies themselves. When the final binary is built the IDE will compile it into one file/bundle.

     

    You'll notice that the AI plugins are usually larger (potentially gigabytes each) than the average plugin (a few megabytes).

      Izaan Ishaq
      Izaan IshaqAuthor
      Participant
      February 11, 2024

      You're right, I  tried to reverse engineer one of the Blace AI plugin and it has PyTorch, OpenCV 3rd party library linked to the ae plugin itself, so I now understand what it will take to develop such a plugin.

      Chris Vranos demotivated me 😞  in your interview with him on your YT where he said how tedious/difficult the plugin dev in C++ for AE is, but still, I will see how much difficult it is for myself xD.


      Anyways thanks James, your brief plugin tuts gave me a solid base to start developing and NT Productions (YT) is great for further deep learning. 

      Mylenium
      Mylenium
      Legend
      February 7, 2024

      All AE cares about is whether it receives the data in a format it understands either via the APIs or using other methods like JSON or copy & paste. The rest is entirely up to you. Most of the tools you mentiuon simply set up a virtual memory host and then convert the data structures dynamically. Whether or how that will work with the Meta tracker is of course an open question, but I can't see why registering a bunch of property streams and shuffling around the image buffers shouldn't work. The same relates to the question of whether the code can be included in the plug-in itself or must be accessed in other ways. I'm sure the terms of use for the tracker code have something to say about this.

       

      Mylenium

        Terms & ConditionsAccessibility statement
        Using the community Terms of use Privacy Cookie preferences Do not sell or share my personal information ad choicesAdChoices