What type of footage are you editing? What camera was used to record this? If it was a mobile phone, you may deal with variable framerate issues. Please use an app called Handbrake (available from www.handbrake.fr) to transcode the footage, using the constant framerate and Placebo quality settings.
Normally a Canon camera would not record in variable framerate, unlike a mobile phone. So that rules VFR out as a source of your delay.
h.264 really is a delivery codec, not an editing friendly codec. Depending on how efficient the files were encoded, your player (in this case Premiere Pro) may have more trouble in playing this back, let alone allowing you to edit the contents.
Let's start with your hardware. What type of computer are you using? Can you provide details?
Have you tried to reduce the playback resolution from Full to 1/2 or 1/4? Does that help?
Another method might be to transcode your footage to an edit friendly codec first, such as ProRes or DNxHD. You can also choose to let Premiere Pro generate proxies from the source clips, so you can do your edit and then render out the final result using the source clips, to prevend degrading the image as a result of transcoding. More details can be found here: https://helpx.adobe.com/premiere-pro/using/ingest-proxy-workflow.html