Poor scrubbing performance even with proxies

I have a highly specced PC. The funny thing is I create proxies for all videos ingested, and the proxies for my A7rii 4K videos scrub very well. However, with lower quality handphone videos from the Huawei P40 Pro (4K30 or 4K60), the proxies lag when scrubbing. The original works better than proxies for scrubbing but it is still choppy. 

Tried both Cineform & ProRes proxies @ 720p.

Proxy info: 

Original video info:

Why is that so and what can I do to get silky smooth scrubbing? 

Ryen 5600

32GB Ram 

SN750 NVME SSD

RTX3060Ti