tweaking audio to viseme pickup

Question

Hi all, I am trying to get more accurate audio to viseme pickup, for a live feed driven character. A lot of the help out there talks about tweaking visemes on the timeline, post record. Even making extra in-between mouths for a closer fit.. but again to be tweaked on the timeline, post record. Other help says try this and that with the mic. No, the Roland Quad-Capture is just fine. As is the computer.

Is there nothing that can be done at the audio detection end to allow for certain vowel sounds to be picked up more than others and certain facial expressions to not be picked up as much. The detection feels a little one size fits all. Can this be tweaked somewhere under the bonnet. A slider that says stay on "Rrr" sounds for 5ms longer for example.

So to paraphrase:

Live feed character

preferring some mouth visemes over others.

How to pre tweak lengths of visemes and or vowels sounds

without a timeline.

Thanks RobG

QTQ Channel Nine

Australia

Dell T7810 16 Core Workstation 64GB DDR4 ECC RAM, 1 x 256GB SSD, 1 x 1TB 7200RPM SATA HDD, Nvidia Quadro K4200 4GB Graphics Adapter, Windows 10 Enterprise x64

wil_li · Answer

Here's another "hidden" way to tweak the lip sync performance. There is a debug parameter that allows you to increase the amount of smoothness in the lip sync computation even further. To tune this parameter, hit Cmd+F12 (or Ctrl+F12 on Windows) to bring up the console panel (next to the timeline), use the console's hamburger menu to pick “Debug Database View”, and then look for LipSyncher::Smoothness. By default, it's set to 1.0, and if you increase it and re-run compute lip sync, you should get increasingly smooth results. For live lip sync, you'll probably need to turn on and off the microphone in the Camera + Mic panel in order for the new smoothness setting to take. In my tests, varying the parameter between 0 and 2 produces a pretty significant range of smoothed results (values greater than 2 tend to be "undersynced" with too few transitions).

That's not the level of control that you've described above, but it may provide a bit of useful tunability. As for more fine-grained control (like holding specific sounds/visemes), that's something that we could definitely experiment with. We will definitely let you know if such controls show up in future pre-releases. They will probably start out as debug parameters (like the smoothness), but if folks find them useful, we'll try to move them into the actual UI.

Sign up

To post, reply, or follow discussions, please sign in with your Adobe ID.

Sign in to Adobe Community

To post, reply, or follow discussions, please sign in with your Adobe ID.