Feature Focus: More accurate lip sync

[Jeff Almasol]

The latest version (3.4) of the lip sync engine improves automatic lip sync and timing of mouth shapes (visemes). Prior versions can still be selected and viseme detection can be tuned in Lip Sync preferences.

Try out the improved lip sync engine in both live/rehearsal and offline (Timeline > Compute Lip Sync Take from Scene Audio) workflows, and let us know how it's working for you.

Are you needing to do fewer corrections to computed lip sync?

Are you able to adjust the Viseme Detection and Audio-based Muting settings (in Lip Sync preferences) to get desired results?

(Use this Beta forum thread to discuss this feature and share your feedback with the Character Animator team and other Beta users. If you encounter a bug while using this feature, let us know by posting a reply here or choosing "Report a bug" from the "Provide feedback" icon in the top-right corner of the app.)

[Frannie@Cartoonfun]

Having run a few tests comparing Ch Beta generation of lip-sync, with existing offline recorded speech tracks that I've used previously with Ch 2020, it feels like the amount of post-compute correction that I need to do in Ch Beta is pretty much the same.

Pre-existing inaccuracies ('quirks' 🙂) are extant. For example, I find that in both Ch 2020 and Ch Beta, 'S' is almost always mistakenly generated when it needs to be an 'F' ... and 'D' or 'L' is generated when it should be an 'M".

It may well be down to my diction, of course... a slight Cornish (Cornwall, UK) burr as opposed to Received Pronunciation.

Anyway, hope the feedback helps (loving the new Beta features overall 🙂)

[Jeff Almasol]

Hi @Frannie,

Would you be willing to share one or more sample audio files that are not working well, and that our research team can diagnose internally? If so, you can send me a direct message with a link to a shared archive (via Dropbox, Google Drive, etc.).

Also, were there any settings in the Lip Sync preferences that improved the results for you?

Thanks.

[ebenheizerm28787053]

Is this lip sync optimized only for english? I am making an Indonesian animation.

[oksamurai]

@ebenheizerm28787053 - while lip sync is primarily optimized for English first, it should work well with a wide variety of languages. If there are certain sounds that don't seem like they work well for you, please let us know - including audio samples is even better!

[Frannie@Cartoonfun]

No probs @Jeff... I've bagged up some stuff and DM'd you the link 🙂 

[scotte57842618]

Yes, we find 'M' gets overused alot. What should be a UH ends up being a M, and S and D for some reason often seem to get stuck on the visme longer

[Timm Schindel]

I'm under the impression that it has improved. At least in the test project I use. It's 8fps and has a reduced amount of visemes.