Yes, with the Marker panel open, ExtendScript-based marker manipulation suffers significant performance issues; thumbnails aren't free. 🙂
So, in my example above, you could 1. save the initial project with workspace, 2. open a different project containing a known (Markers panel-free) workspace, 3. fiddle with markers, then 4. re-open the initial project, and 5. close the different project.
One much less complicated approach: Your panel could also pop a confirmation dialog (hopefully, containing a 'never remind me again' checkbox) that says something like "performance will suffer if you do this with the Markers panel open; continue?"