decreasing duration of a time-design layer increases effect/wiggle speed

I have a layer with time-design responsive intro/outro applied to it. the layer also has a distortion applied to it with the evolution being set to the time expression so its continuous. i also have a wiggle animation applied. my issue now is that if i increase/decrease the duration of such a precomp, the wiggle and effect evolution speeds increase drastically. the initial layer duration within the precomp is 16 seconds, so now if i add one "instance" of the precomp and change its duration to, lets say 4 seconds, the wiggle animation plays 4 times faster.. what do i do? i've even tried adding a third time-design protection  to the entire middle part (starting at the end of intro and ending at the start of the outro time-design protection area), but that also didn't work.. is there any way to fix that?