Highlighted

Feature Focus: More accurate lip sync

Adobe Employee ,
Aug 14, 2020

Copy link to clipboard

Copied

The latest version (3.4) of the lip sync engine improves automatic lip sync and timing of mouth shapes (visemes). Prior versions can still be selected and viseme detection can be tuned in Lip Sync preferences.

 

Lip Sync preferences (3.4).png

 

Try out the improved lip sync engine in both live/rehearsal and offline (Timeline > Compute Lip Sync Take from Scene Audio) workflows, and let us know how it's working for you.

 

Are you needing to do fewer corrections to computed lip sync?

 

Are you able to adjust the Viseme Detection and Audio-based Muting settings (in Lip Sync preferences) to get desired results?

 

 

(Use this Beta forum thread to discuss this feature and share your feedback with the Character Animator team and other Beta users. If you encounter a bug while using this feature, let us know by posting a reply here or choosing "Report a bug" from the "Provide feedback" icon in the top-right corner of the app.)

TOPICS
Feedback

Views

238

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more

Feature Focus: More accurate lip sync

Adobe Employee ,
Aug 14, 2020

Copy link to clipboard

Copied

The latest version (3.4) of the lip sync engine improves automatic lip sync and timing of mouth shapes (visemes). Prior versions can still be selected and viseme detection can be tuned in Lip Sync preferences.

 

Lip Sync preferences (3.4).png

 

Try out the improved lip sync engine in both live/rehearsal and offline (Timeline > Compute Lip Sync Take from Scene Audio) workflows, and let us know how it's working for you.

 

Are you needing to do fewer corrections to computed lip sync?

 

Are you able to adjust the Viseme Detection and Audio-based Muting settings (in Lip Sync preferences) to get desired results?

 

 

(Use this Beta forum thread to discuss this feature and share your feedback with the Character Animator team and other Beta users. If you encounter a bug while using this feature, let us know by posting a reply here or choosing "Report a bug" from the "Provide feedback" icon in the top-right corner of the app.)

TOPICS
Feedback

Views

239

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Aug 14, 2020 0
Participant ,
Aug 23, 2020

Copy link to clipboard

Copied

Having run a few tests comparing Ch Beta generation of lip-sync, with existing offline recorded speech tracks that I've used previously with Ch 2020, it feels like the amount of post-compute correction that I need to do in Ch Beta is pretty much the same.

 

Pre-existing inaccuracies ('quirks' 🙂) are extant. For example, I find that in both Ch 2020 and Ch Beta, 'S' is almost always mistakenly generated when it needs to be an 'F' ... and 'D' or 'L' is generated when it should be an 'M".

It may well be down to my diction, of course... a slight Cornish (Cornwall, UK) burr as opposed to Received Pronunciation.

Anyway, hope the feedback helps (loving the new Beta features overall 🙂)

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 23, 2020 1
Adobe Employee ,
Aug 24, 2020

Copy link to clipboard

Copied

Hi @Frannie,

 

Would you be willing to share one or more sample audio files that are not working well, and that our research team can diagnose internally? If so, you can send me a direct message with a link to a shared archive (via Dropbox, Google Drive, etc.).

 

Also, were there any settings in the Lip Sync preferences that improved the results for you?

 

Thanks.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 24, 2020 0
Contributor ,
Aug 25, 2020

Copy link to clipboard

Copied

Is this lip sync optimized only for english? I am making an Indonesian animation.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 25, 2020 0
Participant ,
Aug 25, 2020

Copy link to clipboard

Copied

No probs @Jeff... I've bagged up some stuff and DM'd you the link 🙂 

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 25, 2020 1
Adobe Employee ,
Aug 25, 2020

Copy link to clipboard

Copied

@ebenheizerm28787053 - while lip sync is primarily optimized for English first, it should work well with a wide variety of languages. If there are certain sounds that don't seem like they work well for you, please let us know - including audio samples is even better!

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 25, 2020 0
Contributor ,
Aug 25, 2020

Copy link to clipboard

Copied

Yes, we find 'M' gets overused alot. What should be a UH ends up being a M, and S and D for some reason often seem to get stuck on the visme longer

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 25, 2020 0
Explorer ,
Aug 28, 2020

Copy link to clipboard

Copied

I'm under the impression that it has improved. At least in the test project I use. It's 8fps and has a reduced amount of visemes.

Likes

Translate

Translate

Report

Report
Community Guidelines
Be kind and respectful, give credit to the original source of content, and search for duplicates before posting. Learn more
Reply
Loading...
Aug 28, 2020 0