I never had issues with modifying Speed / Duration on my 2020 Macbook Pro (M1 chip) using the Intel version of Premiere Pro. Once updating to the 15.4 (Silicon) all of my playback and export became shaky, jittery, rubber banding, and choppy (as if frames are being dropped) on any clips that use Speed / Duration edits. After exporting, the same clips have the same issue in the final exported file. I switched back to the Intel version and everything seems fine now.
Note: With 15.4, I tried uninstalling, reinstalling, replacing clips, starting new projects, sequences, render/replace, you name it, I tried it. The only fix was reverting back to the Intel version.
Crash, Error or problem, Freeze or hang, Hardware or GPU, Performance
We're sorry about this. We can get this checked. Are you experiencing this issue with any specific media type or is it with all of them? Have you tried creating a new project & importing the original project to check if it's working properly in that?