Lip syncing only recognizes a few mouths

Report · Jul 05, 2020

When in Record, I only see the puppet's mouth change to a number of mouths. Neutral, smile, and surprised are the only mouths shown when I'm recording my face and I'm not sure why. Any ideas as to why this is happening? Thanks! If you need any more information please let me know. I love this program and hope to improve my puppet

Report · Jul 05, 2020

UPDATE

After recording a bit, I do see a few of the different mouths but its not as fluid as I'd like it to be. The puppet's mouth doesn't seem to match what I'm saying in the camera view. Any help with this?

Report · Jul 05, 2020

Neutral is the default. Smile and Surprised come from the webcam and face behavior. The rest come from audio processing and the lipsync behavior. If audio quality is low, then it will do a less good job.

Report · Jul 06, 2020

Is there anyway to improve my audio quality? I have a mic connected and I'm speaking directly into it, but the lip-sync is still off a bit. My mic is not too old so I dont think its the mic itself.

Report · Jul 06, 2020

Could you show a screen shot of the timeline panel showing the visemes it picked for saying something with lots of different sounds? (The quick brown fox jumped over the lazy dog, or something like that.) I don't know how to fiddle the settings sorry beyond the obvious, but it is definitely not perfect. Seeing a waveform of an audio recording alongside the visemes it picked might help identify whether "that is as good as it gets" or not. (Note: you can edit the visemes afterwards if recording, but live performances that is not an option, and its time consuming.)

Report · Jul 06, 2020

I'd like to do live performances with this so trying to get the best results. Here is the timeline for the phrase you recommended.

I did notice that before saying anything, the mouth started to change. Not sure what triggered this. I turned off everything around my mic that could have caused some sound aside my own voice and the mouth was still changing for some reason.

Also noticed this lip-sync option. Not sure if adjusting it will help with getting the best result.

Report · Jul 06, 2020

Certainly try the lipsync sensitivity. You might try adjusting microphone sensitivity too. For example, the silence seems to have a fairly significant bar at the bottom (like lots of background noise from a computer fan or similar). Also the max peaks look like they might be being cropped in places (like you are talking too loudly causing audio distortion). Hard to be sure -it might be fine, but whenever I see a solid white section where its all hitting the top I worry that the audio is being "clipped". You could try editing the voice clips in Audition and run "noise reduction" over them, then do lip sync from file. If it works better then it would contribute to sound quality being a factor.

In case you did not know, audio is basically wave forms. Inside the computer they are turned into a series of numbers (e.g. from 0 to 10,000). If no noise, then when you don't talk it will be flat at the bottom (all values of zero). So the fact that there is a solid bar along the bottom means its not silent. Clipping is when the audio goes so loud that it exceeds the 10,000. The computer says "wow, 12,000 - let me clip that to 10,000 because that's as big as I can handle. The numbers is not really 10,000, just an example. The end result is the audio sounds distorted. Its like someone spoke too loudly into a microphone - it does not sound good afterwards.

What that all means is if you zoom into the audio track, you want the line to go from near zero up to the max value, but not beyond it on either edge. You want nice clean lines going up and down with out any clipping. And you want a flat line when nobody is talking (no noise). "Noise reduction" in Audition can help reduce noise, but there is not solution to clipping. You just need to make the audio better quality by getting the levels correct.

But it is finding a good mix of visemes - that is one question answered.