I found a feature request on Uservoice to speed up the hover state. I'd recommend upvoting it there for the product team to see.
There was a comment there to try setting the hover transition to 0. You could try that.
I wonder if it's a lag in the actual triggering, because any effect is immediately visible. I guess it's the animation duration which gives you the impression of a slow effect. Try setting the time to 0 seconds (and no easing).
[EDIT] Oops, I see Myra also mentioned the same solution 🙂 I just have a lag of 40 minutes...