Can you upscale video too much?

I am planning to make a video which involes clips used, some of which are 1080p, but others oonly 720p, and I have Instant 4K for Premiere Pro, allowing you to instantly upscale 1080 to 4k so you can export in 4k, however I don't know about 720p, as I would've thought upscaling it all the way to 4K would end up looking worse than before, or make no difference at all? Is this true, is there a point where it will just make no difference or make it worse?