Character Lip Sync. Is this as good as it gets?

[Jon Chambers]

Hello,

I've just finished processing 10 minutes of audio and the lip sync feels... off.

Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.

I don't want to sound ungrateful. If I had to lipsync 10 minutes worth of audio and had to do it in real time, my results would be much worse.

Though how do I polish this? Yes, I know I should do a manual pass, but is there anything I can do before that? Is there any filters I can run the audio through to give the computer a better read? Should I re-record with a webcam, so that the computer will have more lip information? Is there a big update coming out in a couple of weeks I should wait for, before re-processing?

I hope I've exported this correctly. Please forgive me if it's missing an important file. I've never exported a character project before.

Also, as an aside, I welcome comments on the actual content and script of my audio if the moderators don't mind a few side comments about politics or employment as an animator. (Moderators, feel free to delete this last paragraph or the word "politics" if I was out of line by saying it.)

[oksamurai]

I don't see a chproj file - could you either upload that or export out a sample video to check out?

[Jon Chambers]

OneDrive says that chproj is an empty file, and refuses to upload it. I'll try zipping it and upload it again. Just a second...

[Jon Chambers]

Fingers crossed, this one will work Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.

[Jon Chambers]

Let me know if you've got it, because I'm working on exporting a sample video as we speak.

[Jon Chambers]

It will take a while to upload, but by the time you see this, it may be finished on this link Microsoft OneDrive - Access files anywhere. Create docs with free Office Online.

[oksamurai]

I could open the chproj but it was missing the wav audio file. Can you share that?

[Jon Chambers]

Sorry about the delay. That's Australian internet speeds for you. Microsoft OneDrive - Access files anywhere. Create docs with free Office Online. It was a very long upload.

[oksamurai · Accepted solution]

Thanks!

So I bet the biggest culprit here is how low the audio volume is - if you open it up in Audition, for example, you can see how quiet the audio is. This means CH has to strain and do a lot more work trying to isolate the sounds. If you normalized or added a hard limiter to this, you should get much more reliable results.

Overall it looks good - the biggest thing I see are some missing visemes every once in a while, which I think the higher volume should fix.

[Jon Chambers]

Ha! I actually did that, but forgot to click "save". Such is life.

[Jon Chambers]

Yes, definitely a noticeable improvement, but still a little shy of perfect. So, the best workflow is a manual pass next? Or am I skipping a step? I notice it struggles to pick up the letter "P" when I say it.

[oksamurai]

There isn't a P viseme, so if you want to trigger a special puckered lip or something in these parts, then yes - I would create a custom mouth that is triggered by the P key and record it in where you need it.

In traditional animation the P often "pops" into the next sound, like the "ah" in poverty - if I test here with your puppet it looks pretty good to me!

[Jon Chambers]

I know that the M or neutral viseme is all that's required, but sometimes I know I said "P" but the puppet has its mouth open the entire time. It's only happened maybe once every couple of minutes, so that's still a good run. I can fix it manually.

Some advice I heard is that getting visemes 100% correct isn't important provided the mouth is open at the correct times and closed at the correct times. (Though, with my puppet, the tongue should be in the right place also.)

No one ever noticed how few visemes Kermet the Frog had.