Visibility of a layer related to sound

[NathGros_BSS]

Hi all,

I've searched the forums a lot on the subject but I can't find a solution to my problem. The closest would be this article but it doesn't match the animation I want to make...

I'm working on an animation for an audio podcast. I would like the thumbnail of the speaker who speaks to appear automatically when he starts speaking and disappears when he finishes.

So I need to link an animation of the layer opacity from 0 to 100 and from 100 to 0 with the audio spectrum. do you think that is doable?

Thank you to all of you !

[Mylenium]

You cannot link anything to an effects internal stuff unless it is exposed as a regular property. That being the case, whatever you have in mind will need to be prepeared by e.g. using Premiere's audio tools to create cuts and markers and reference those as per the expressions you already found. Point in case: AE is neither an audio nor an actual video editing tool. It's really just meant for graphics and you have to include other tools in your workflow.

Mylenium

[Dan Ebberts]

Assuming you have converted your audio to keyframes, something like this could possibly work, although it will probably be slow (unless maybe you convert the expression to keyframes once you're happy with how it works):

threshold = 5;
fadeTime = .5;
L = thisComp.layer("Audio Amplitude");
audio = L.effect("Both Channels")("Slider");
tIn = L.inPoint;
tOut = L.outPoint;
for (i = L.inPoint; i <= L.outPoint; i += thisComp.frameDuration){
  if (audio >= threshold){
    tIn = i;
    break;
  }
}
for (i = L.outPoint; i >= L.inPoint; i -= thisComp.frameDuration){
  if (audio >= threshold){
    tOut = i;
    break;
  }
}
if (time < (tIn + tOut)/2){
  linear(time,tIn,tIn+fadeTime,0,100);
}else{
  linear(time,tOut-fadeTime,tOut,100,0);
}
 

[NathGros_BSS]

Thank you Dan Ebberts,
Yes I did convert audio to keyframes.
I applied your expression to the opacity of my layer, I have a fade in and fade out but only at the beginning and end of the composition (the ajustement of the fadeTime work well).

We would say that it does not react with the audio keyframes even by adjusting the threshold...

[Mylenium]

Rick answered your question and that's why it doesn't work. Spoken language has no distinct peaks for any expression algorithm to determine anything. As Rick and I already said you have to actually chop up your file or create markers so the marker times or in- and out-points can be used.

Mylenium

[Rick Gerard]

You can not get an expression to recognize a voice, but you can get an expression to recognize when audio levels move above the base noise level or look at layer markers. This means you would have to separate the audio into two tracks, one for the speaker and one for the host, or you would have to add an audio marker every time the speaker starts to talk and, another one when they stop. By far, the simplest solution is to split the audio track so the speaker is only on one track, then add layer markers or just all layer markers when you want the opacity to change. 

To tell you the truth, if the podcast is longer than about 1 minute, I would do the whole thing in Premiere Pro  by just making cuts in the layer containing the speakers' image and using the keyboard shortcuts to add fade in and fade out. You could do all of the cutting in real-time while the clip was playing by just hitting a key, then jump back into the timeline and delete all of the parts where the primary speaker wasn't talking. 

[NathGros_BSS]

Thank you Rick Gerard,

There will be 3 speakers and a host. I will receive each track individually, cleaned.

I'm going to have 3 podcasts over an hour each, so my effect needs to be automated.

By converting each audio tracks to keyframes, and with a low threshold, after effects should easily recognize when the speaker is speaking. You just have to manage to tell him to launch a fadein and a fadeout at the right moment...

[ShiveringCactus]

I've done a fair bit with audio, keyframes and After Effects.  I'm not sure from your responses what issue is that you're still having.  I think you're saying that as a speaker starts speaking you want their "symbol" to appear and stay on screen until they stop speaking, at which point you want it to disappear.   If that's the case, don't you just need the "on/off" threshold to be very close to each other, that way as soon as there is noise the symbol appears?

If I'm following correctly that and given that expressions run each frame, I think you need something simpler and more absolute:

var vol = thisComp.layer("Audio Amplitude").effect("Both Channels")("Slider");
if (vol > 0) {
	100
} else {
	0
}

If there's any volume, show the symbol, if there's none, hide it.  

I'd also be tempted to have each audio and image in a separate precomp, which makes it easier to run the Convert audio to keyframes.

[Rick Gerard]

The problem with that approach ShiveringCactus is that every time the speaker pauses or takes a breath, the graphic or image will blink, there is no fade, and it's going to be a mess. For that idea to work at all, you're going to have to look forward and backward far enough to compensate for normal pauses in a person's speech pattern and the breaks between words, or you're going to have to pre-process the audio, so any background noise will be eliminated between pauses in the audio track where the other speaker is talking.

Somebody, sometime, is going to have to go through the recording manually and define the breaks. 

[NathGros_BSS]

Hey ShiveringCactus,

Thanks for the expression. This works pretty well because the person who is going to edit the audio mutes the audio when the person is not speaking. Even when the speaker pauses in a sentence, there is still background noise that keeps the level above 0 db. So the thumbnail remains visible.

[Rick Gerard]

To write an expression that looks for empty spots in an audio track, you are going to have to pick a time value that is long enough to prevent all normal pauses a person makes when they are talking from turning off the layer while at the same time calculating the start and end time of that pause and creating a transition instead of a cut from 100% opacity to zero. Even if that could be done, the expression would be recursive and extremely slow to render. 

If you drop your images and your audio tracks into Premiere Pro, expand the audio tracks enough that you can see the levels when the different people are talking, then use Ctrl/Cmnd + K to make a cut every time somebody else talks; you should be able to trim an hour-long podcast in just a few minutes manually. You can then use the Up and down arrow key to jump from cut to cut, delete where you don't want to see the image, then select all the sections in the timeline, and press D to add the default Fade In and Out transition to every selected section. You could even use the L key to playback the sequence in real-time or speed it up and just press Ctrl/Cmnd + K to make a cut every time you want the photo to change. I can't imagine that the entire process would take more than 15 or 20 minutes. 

[ShiveringCactus]

Just to clarify - I agree with @Rick Gerard that is the right approach.  But I think what we have here is a classic case of "wanting it done quick or good".  The more time invested will produce a better result.  But podcasts are rarely just one-offs, so I think @NathGros_BSS is looking for a workflow that's not going to feel like a pushing a rock up a hill by episode 10.

One idea: I've been stuck on a LOT of Teams meetings and I noticed that while everyone's picture (or initials) are always on screen, (depending on the software used) they light up or have a circle appear around them when speaking.  Maybe the answer is to replicate that style, where volume produces an effect, without the speaker's image having to disappear completely?

NathGoss_BSS - if there is background noise, you could always adjust the line 

if (vol > 0) {

So that the > value is slightly above 0 - maybe link it to a slider even.