Let me guess what is problem. Your file is 300 ppi while image that you are placing is 72, for example. When you have situation like this, Photoshop will silently change Resolution to match document resolution and in your case (as predicted above) it will enlarge file 4 times (maybe and more what depends on resolution).
Check both document and file resolution
Change either file or document resolution with Resample turned off in the Image Size dialogue. You should match resolutions.
Check document resolution and remember.
Double click on Smart Object thumbnail to edit Smart Object.
Go to Image > Image Size, turn off Resample and change Resolution to match document resolution.
Save changes and close tab with Smart Object content.
A great answer Bojan. Image size and resolution was the one thing that new members of my old camera club struggled with. To be fair, the people stetting the rules for competition entries added to the confusion because they'd state that a digital entry had to be a maxiumum of 1024 x 768 pixels at 300dpi. So I suspect that the people setting those rules were as confused about image size and resolution as our new members.
I'll put my hand up to making my own mistakes. I used to get upset that the images that looked great on my computer screen were faded and unsaturated when viewed via the club's projector. Of course I was submitting my entries as Adobe RGB, and the projector was set up for sRGB as clearly stated in the cometition rules. But who the heck reads instructions? Girls do, using the same genetic advantages that lets them find things on shelves, and actually listen when people tell them things. I am absolutely not going to say anything about girls and maps, because I know how dangerous that could be! 😞