Audio Hardware Causing Playback Stuttering???

So I was having serious playback issues with Premiere Pro lately.  ProRes and H.264 4k video both playing terribly at 1/2 resolution.  I followed all the tips Adobe recommends (clear cache being basically the only one beyond lowering your playback quality).  The only thing that fixed it was I found multiple references to Playback issues being related to Audio Hardware.  

So I switch my Audio Out (in Audio Hardware in Premiere) to one of my monitors (actual PC monitor).  Now my footage is playing smooth as butter.  Previously I was sending audio out to my Motu M2 audio interface.  That interface sends audio to my Studio Reference Monitor speakers and Reference Monitor headphones.  It also is the IN interface for my XLR microphone for voiceover recording.  

Now I found reference to this fix as far back as 2019 (I bet I'd find it further if I kept looking).  So what is going on here?  Does Adobe not consider this a bug?  My monitor speakers are AWFUL, but that seems to be about the only way I get Premiere to actually work as it should.  So do I have to pick whether I get to hear quality audio (to confirm it is even worth using) OR quality video playback?  

And please don't try and pass the buck to Motu on their audio interface.  I truly doubt everyone who has run into this is running the exact same Audio interface hardware.