Writing script to automate lip-syncing in Premiere for animation series

[tomandre]

Hello all,

I am a veteran PPro/AE user and beginner programmer attempting to write a script to automate the lip-syncing tasks I do within an animation series.

Workflow

This project involves video clips that can stretch to 30+ minutes, making placing the required mouth images over each frame of footage a very numbing, tedious process. In short, I'm given a Premiere Project that contains near complete footage, with the only thing missing being the mouths for each character onscreen. We are provided the correct mouth images - in .PNG format - within a separate folder.

I then go through and place the correct mouth images over each character in my footage, with the particular image used being dictated by whether the character is speaking. Luckily, I don't need to move the mouths around using any of the Transform properties or keyframe anything...they are already saved with the correct dimensions and placement, so all I'm doing is dragging-and-dropping the correct mouth image over my footage, one frame at a time. However, placing images of one frame length for each and every frame can quickly get tedious when you're working with shots that can be 30+ seconds, in an animation project that runs over 30min.

As far as the variety of our mouthsets goes, we use three main images to cover each character speaking. These images are divided as such:

• Closed Mouth (character isn't speaking) - usually designated as FILENAME-C.PNG
• Mid Mouth (character's mouth is halfway open) - usually designated as FILENAME-M.PNG
• Open Mouth (character's mouth is fully open) - usually designated as FILENAME-O.PNG

Quick visual example of this:

Scripting Solution

Now, for the scripting goodness...I'm trying to write a script that would automate this workflow. The idealized, script-assisted workflow would look as such:

1) I set In and Out points on the timeline over the footage I want to automate mouth placement. This keeps it contained to just one shot and one corresponding mouthset, instead of the entire timeline.

2) I launch the UI/panel for my script

3) I drag and drop the three main images I need placed into the panel. The script would ideally know which mouth to use based on the filename (FILENAME-C.PNG to designate a mouth that's closed, FILENAME-M.PNG for a mouth that's midway open, FILENAME-O.PNG for a mouth that's fully open, etc).

4) I hit Run.

5) The script looks through the specific frames I've selected via In and Out points. For each frame where the underlying audio is at a minimum (-260dB to -30dB), it pastes the closed mouth file (FILENAME-C.PNG). For each frame where the audio is between -30dB and -10dB, it pastes the mid mouth file (FILENAME-M.PNG). For each frame where the audio is between -10dB and 0dB, it pastes the fully open mouth file (FILENAME-O.PNG).

Now, as far as how to specify this in coding?

I have absolutely no clue. This is where I turn to the community for guidance and suggestions.

My understanding of Javascript is pretty minimal but I can crunch through the basic coding and tutorials and learn it quickly (I did some programming in C/C++ many years ago; it wasn't too hard for me to pick up, nor have I had difficulty with HTML or CSS).

From my searches, I believe that I would need to build the basic panel in HTML 5 with some buttons that trigger the scripting. Once I have a working panel/UI, I can start to communicate with Premiere using ExtendScript.

I've seen examples of this done in After Effects, so I know it can be done (see mamoworld's very impressive Auto Lip-Sync ​), but I have no idea how to pull this off inside of Premiere. Most importantly, the part where I tell the program to look at the underlying audio and paste a one-frame image file based on the audio clip's volume is one thing I'm very uncertain about.

Any ideas, help, suggestions or advice about what to do?

[e_d_]

Hi,

AFAIK Premiere's HTML/JS capabilities do not yet provide any means to analyze audio levels, which means you would have to leverage C++ to do what you want to accomplish. Inserting clips on the timeline is entirely possible though.

[tomandre]

Hi Eric,

Just knowing that alone is a huge help. Thanks for the insight!

So audio volume might not be the correct metric to base my script off of. I was thinking a good tweak to the script might be to just tell it to place the mid mouthsets on any frame that has an audio clip on a specific audio track (say, Audio Track 2). I can easily move all dialogue tracks down to Audio Track 2, and if I can get the script to recognize "hey, there's an audio clip in Audio Track 2 at this frame, I'm going to paste this image for one frame on Video Track 2", that would greatly help.

Does that seem more feasible to you? And if so, what scripting commands would you use to tell the script to look at specific tracks and insert a file? Currently going through the Premiere Scripting Guide, so if any of my questions are ill-worded please forgive me.

[Bruce Bullis · Accepted solution]

One Tough Thing™ = Your use case spans two of PPro's APIs, and is a perfect fit for neither.

There's no way for a panel (built using ExtendScript) to analyze audio, and there's no good way for an audio effect plug-in (built using C++) to insert media onto tracks.

It'd be possible (with lots of work on your part) to (from a panel) get the path to the actual audio media, analyze it using [some external tool], then have the panel act on those results.

[still thinking]

[Bruce Bullis]

> ...what scripting commands would you use to tell the script to look at specific tracks and insert a file?

See PProPanel's randomizeSequenceSelection() for walking across track items, and insertOrAppend() for insertion.

[tomandre]

Thank you so much Bruce! I've been devouring Javascript tutorials the past few days and it seems like the second idea (using the script to look at tracks and insert a file) will be a more realistic solution. Appreciate the PProPanel template you've linked me to, that resource really helps! In the meantime I'll try to mock up something in Javascript with a programmer friend who is far better at this stuff than me.

When I've got a script worth looking over, could I post it here for feedback? Or would you recommend I post it elsewhere (the SDK forum, Github, etc)?

[Bruce Bullis]

You can post here if you want broad feedback, and/or send it to me directly; b b b at adobe dot com.

[Bruce Bullis]

Eric is correct, and...have you tried Character Animator?

[tomandre]

Hi Bruce!

I actually have tried Character Animator but I haven't been able to successfully integrate it into our workflow. We receive Premiere projects of substantial length and several hundred different shots & character mouthsets. I wasn't able to quite figure out how to import all of this into Character Animator and make it work in a way that yielded significant editing gains (the setup alone quickly started to get hairy). Most tutorials I found demo'ed use cases of 1-3 characters and corresponding backgrounds, but if there are any resources/tutorials for doing larger projects, I would love to check them out.

Eric's comment has got me thinking of ways to scale down my script from its ambitious goal of placing mouthsets via audio volume, to just placing the mid mouthsets on any frame that has an audio clip on a specific audio track (say, Audio Track 2). I can easily move all dialogue tracks down to Audio Track 2, and if I can get the script to recognize "hey, there's an audio clip in Audio Track 2 at this frame, I'm going to paste this image for one frame on Video Track 2", that would be a massive help to my editing...and my wrists, haha.

If any of this seems feasible to you, let me know. You guys rock!