Audio Sync + Transcript

[Il Cacciatore di Tool]

Hi Dave, hi community.
There's an issue in new audio+transcript labial sync.
Note: I used an Italian audio (1 minute long) so .SRT is in Italian.
If I try the syncing with audio only, all the audio file is synced but when I use the audio+transcript, the end of the audio file (10 seconds) is not synced and no visemes appear in the timeline.

It's a strange behavior because only the last 10 second are not synced, and the result of the first 50 seconds is drastically better than the sync of audio only.
Have a nice day.

[CoSA_DaveS]

Thank you for the report. Adobe is closed today (Friday); we'll get back to you next week.

Do any markers with error info appear in the timeline?

[Il Cacciatore di Tool]

Hi.
No markers with error in the timeline.
I tried again yesterday. Same result.

Have a nice day.

[dtull-adobe]

Hi,

Thanks for giving it a try! I think this may be a similar issue to what I described in a response here: https://community.adobe.com/t5/character-animator-beta-discussions/feature-focus-transcript-based-li...

The code tries to align sounds to phonemes and sometimes it does that a bit to aggressively so it runs out of text before it runs out of audio. It is possible this is exacerbated by it being Italian, though it does happen sometimes even for English.

One thing I might have to check is how the backend handles accented characters. If it is skipping them instead of mapping them to an unaccented form (would have to go digging in the code to confirm), it might make the effect worse. One way to test that would be to replace such characters with unaccented ones and see how the result compares.

If you'd like to share the audio and transcript, I can try to take a closer look, but I think I have a rough idea of what's going wrong. Thanks for the report! It's good to know that even on a language we haven't tested, there's at least some promising results (even if there's also some issues to sort out)!

Dan Tull

Adobe Character Animator Team

[Il Cacciatore di Tool]

Hi Dan.

I tried to remove all accented characters.
I also tried to modify the .srt file. It was in UTF-8 BOM format and I saved it in UTF-8 and I put the text of every time block in a single row,

Mission failed.

How can I share audio and transcript?

Thanks for your help.

Have a nice day.

[dtull-adobe]

I sent a private message with some instructions. 

Thanks,
DT

[Il Cacciatore di Tool]

Just sent you the link
Thank you.

[dtull-adobe]

Ok, I think I see what's up and the good news is that it fits with another improvement I already want to make.

It does seem to be the problem I suspected. Note the "M" sound between phrases at 14-16 seconds and another in the 36-38 second range. In the viseme track, there are visemes being aligned to those sounds but no corresponding text so those visemes are "consumed" and it runs out before it gets to the end.

I need to look at the alignment that's happening, but if I use a debug option to turn off the coalescing of SRT segments (so it runs each one separately), I get a better result for the remaining words.

Another way that doesn't require access to hidden knobs is to actually insert transcript content for some of the voice parts that aren't represented in the SRT. So for example, if I insert these in the SRT (the Ch parser doesn't require the numbering, so I didn't change other numbers or add ones for these), I also get a better result. I still lose the Ciao at the end, but the 49-56 second range does get visemes.

00:00:14,560 -> 00:00:16,240

Mm m m mm m m m m m m

00:00:36,840 -> 00:00:38,360

Mmm m m m m m m m m m

This also raises another area we know is a bit of a weakness and that is that this scheme doesn't really handle vocalizations other than words very well.

Thanks for sharing your audio/text, this was a helpful example to look over!

Ciao!

Dan Tull

Adobe Character Animator Team