Feature Focus: More accurate lip sync

Report · Aug 14, 2020

The latest version (3.4) of the lip sync engine improves automatic lip sync and timing of mouth shapes (visemes). Prior versions can still be selected and viseme detection can be tuned in Lip Sync preferences.

Try out the improved lip sync engine in both live/rehearsal and offline (Timeline > Compute Lip Sync Take from Scene Audio) workflows, and let us know how it's working for you.

Are you needing to do fewer corrections to computed lip sync?

Are you able to adjust the Viseme Detection and Audio-based Muting settings (in Lip Sync preferences) to get desired results?

(Use this Beta forum thread to discuss this feature and share your feedback with the Character Animator team and other Beta users. If you encounter a bug while using this feature, let us know by posting a reply here or choosing "Report a bug" from the "Provide feedback" icon in the top-right corner of the app.)

Report · Aug 23, 2020

Having run a few tests comparing Ch Beta generation of lip-sync, with existing offline recorded speech tracks that I've used previously with Ch 2020, it feels like the amount of post-compute correction that I need to do in Ch Beta is pretty much the same.

Pre-existing inaccuracies ('quirks' 🙂) are extant. For example, I find that in both Ch 2020 and Ch Beta, 'S' is almost always mistakenly generated when it needs to be an 'F' ... and 'D' or 'L' is generated when it should be an 'M".

It may well be down to my diction, of course... a slight Cornish (Cornwall, UK) burr as opposed to Received Pronunciation.

Anyway, hope the feedback helps (loving the new Beta features overall 🙂)

Report · Aug 24, 2020

Hi @Frannie,

Would you be willing to share one or more sample audio files that are not working well, and that our research team can diagnose internally? If so, you can send me a direct message with a link to a shared archive (via Dropbox, Google Drive, etc.).

Also, were there any settings in the Lip Sync preferences that improved the results for you?

Thanks.

Report · Aug 25, 2020

Is this lip sync optimized only for english? I am making an Indonesian animation.

Report · Aug 25, 2020

@ebenheizerm28787053 - while lip sync is primarily optimized for English first, it should work well with a wide variety of languages. If there are certain sounds that don't seem like they work well for you, please let us know - including audio samples is even better!

Report · Aug 25, 2020

No probs @Jeff... I've bagged up some stuff and DM'd you the link 🙂

Report · Aug 25, 2020

Yes, we find 'M' gets overused alot. What should be a UH ends up being a M, and S and D for some reason often seem to get stuck on the visme longer

Report · Aug 28, 2020

I'm under the impression that it has improved. At least in the test project I use. It's 8fps and has a reduced amount of visemes.