Different files may have drastically different codec applications/settings, and those things matter a heck of a lot. For example, with H.264/long-GOP format codecs, how long the groups of pictures are makes one massive difference in the computational load on the computer.
If one clip is staying with say 15 as the outside number, and the other is 30+, and maybe even using "partial" i-frames (complete frames) spreading out the computational workload to a vastly higher number of 'frames' simultaneously ... that latter clip is going to be horrendous to work with.
And the first clip will be no problem. "But both are H.264!" ... yea. But very, very different files.