I'm just about to start developing illustrator on mac.
Is it possible to use cocoa framework to design the UI instead of ADM?
Then my plug-in code will be mixed with C++ code and objective C++ code. I don't see this from the sample code, can anyone tell me whether it is possible?
I can't say for sure, but since I know from experience it's possible to mix in other GUI toolkits, I don't see why not. You won't be able to use anything but the ADM for palettes, but for just dialogs that pop up? I can't think of a reason why not.
It'll be your headache to parent them appropriately though. AI provides a way to get the root window for the application, but you may have to do some work to find the appropriate child of that window to use as the parent for your dialogs.
If you can do a Cocoa window in a Carbon app in general, then this should work in Illustrator without too much difficulty. If you haven't considered it, you may want to look at some cross-platform UI toolkits: Qt, wxWindows, gtk, etc.