Remove everything, but leave drums

Having listened to your original, and your attempt to isolate the drums, I don't think you'll actually get much better results than that - they really aren't that clear in the original (although that might be because it's an MP3, which is never a good starting place for this sort of exercise). If what you've got is reasonably accurate, then it's not much of a drum track anyway, and all you really need to do is concentrate on the BD beats. The chances are that the video cuts are simply in the wrong place; the normal way to do this is to sort out the sound first so that it's continuous and then cut the video to it (preferably on the beat). Back in the day I did loads of these sorts of videos, and because we were doing it with tape editing, this was, in practice, the only way to proceed if you wanted any sort of a decent result.