I can't speak to why you're having these slowdowns but your theory on images being faster than text does not hold up.
A full-page JPG can be 5 MB.
A full-page TIF image (my usual scanning output) can be about 20 MB.
A full page of text can be about 60-150 kb.
I suspect that something else is slowing things down. I would guess it's something that needs to parse the text before you can see the text, but I do not have a clue as to what that could be. Hopefully, someone else may be able to shed some light in that direction.