When I make a smart object in Photoshop want to edit it, it always opened in a new window. When it updated it asked me what to use to open the smart object. I pressed illustrator by accident. Now I cant change that preference back to Photoshop
Go into your Windows system settings and change file associations for .psb to Photoshop. Raster smart objects will be in the .psb format. You set .psb to open in Illustrator when you clicked that dialog.
Not that vector smart objects should open in Illustrator!
That would be normal for Vector Smart Object for Photoshop does not support vector image files. AI does. Raw Smart object will open in ACR for photoshop uses ACR to support Camera Raw files. Other smart objects will open in Photoshop