Well, all your clips have FX all over it in Premiere, which could mean that you are messing up framerates just by that. this seems liek the classical misunderstanding where you simply already moved to far ahead in Premiere and nearly finished the project instead of just leaving it a rough cut. Start by actualyl creating a version of the project with the effgects stripped down and see if it makes a difference. Likewise, double- and triple-check your footage interpretation, sequence settings and comp settings. Somewhere there could still be a misinterpretation.
You answered your own question: You cannot use speed ramps in premiere and then expect everything to line up perfectly in AE. From normal quantization issues to simple differences in the algorithms it is inevitable that slight variations creep in. you have to make up your mind where you actually want to do the speed-up. Otherwise you will have to render the Premiere timeline to an actual clip to bake in your existing speed ramps.