It sounds like you want your VB window to stay modal? Or maybe semi-modal with Illustrator? If that's right, there's no easy way to do that with a VB project I imagine, but I'd assume you'd have to find some way to link your VB project's window with Illustrator's frame window if you wanted something like that to work. If it's not parented off Illustrator, Illustrator is going to jump up on top of your window.
Even if you do manage it, I'm not sure it'll work anyways. Illustrator has a very odd windowing system to facilitate the palettes & their quasi-modal functionality. It may not be possible to get an external application -- even one launched from within Illustrator -- to work properly within that framework. At least, not without a heck of a lot of hacks.