Hi. Did you do the transformation after you applied the filters? To me, it looks like when you increased the size of the pixels, the computer tried to resize the effect, but it's not very good sizing anything up. There are a fixed number of pixels and when you bring the cat image up in size, you're really simply stretching pixels, which then Photoshop need to try to resolve by essentially guessing how to rebuild them. If I was doing this, I'd try first - before ever bringing the cat into the background - the Neural Filter named Super Zoom. And, if it's being resized a lot, I'd do it in smaller increments (convert for Smart Filters, first). THEN, when the cat's the right size, move it into the new image. THEN run your filters. Hope this helps.
... View more