That is pretty much expected when using this stack technique. AE layers have no thickness and there is always going to be some point where they are almost perpendicular to the camera and effectively disappear. You have to put in some extra work and e.g. cover it up with a matching solid strip that you fade in for those few frames when it's needed.