Why is scaled 4K footage grainy when I zoom in?

I often film interviews in 4K so I can punch in for a close up when needed. I've dropped loads of 4K footage in a 1080 timeline and half the 4K shots are really grainy when I punch in for the close ups and half aren't. I cannot for the life of me work out what is causing this though. I think it has something to do with scaling the footage down to size. I always select scale to frame size as opposed to set to frame size, but I think both of these lose the resolution. It appears when I manually scale, this is what gives me crystal clear 4K.

Has anyone experiences this?