Simple question im sure im overcomplicating concerning masking

Hello and thank you in advance. My question is when i have a picture and embedded another picture on top of it. My top picture always appears as a smart object which off course leaves me incapable of making adjustments to the top picture unless i first Rasterize smart pic. However it seams as though recently when i do this and attempt to brush over the top picture my brush strokes are super pixalated and difficlt to do percesion strokes. This is something ive only dealt with recently and im trying to figure out if i possibly changed some setting? When i reffered to the strokes being pixilated i included a pic at the bottom. Thank you.