"I was working with smart objects in photoshop and sudently it opens a window asking me which application"
Did you try to open the Smart Object or not? If you tried to open the Smart Object how was the Smart Object Layer created in the document you were working in? I have only seen a dialog open ask what application when I try to open some file type that my Windows system has no application associated with. It sound like the Object file type was not a file type the Photoshop supports so Photoshop through the file to your OS to open and you OS had no application associated with the file extension. I would have thought if it was a Vector smart object like a .svg or .ai that if you had Illustrator installed the it would be associated with .svg and .ai the Illustrator would open. I do not install Illustrator. If I open a vector smart object and its and .svg Chrome displays the images. If its and .ai note pad opens it as a text file. No dialog asking what application. So why did the dialog happen to you and what type of file did illustrator one when you chose illustrator.
Perhaps you should as in the Illustrator forum. You may also want to open the .ai file in notepad to see it there are any paths in it or it its all pixel data. Photoshop is a pixels edited it support a few vector type layers. I do not install AI I doe not want to learn how to use an other complex Adobe application.