I don't need it at runtime, but I was getting the same error from AE about my plugin being invalid. I essentially added the .dylib file to the list of dynamic libraries under the "Link Binary With Libraries" heading in the "Build Phases" tab of my bundle target. Then I added the path to the .dylib file under the "Library Search Paths" setting under the "Build Settings" tab of my target.
I do have a header file for the .dylib, and I placed it in my plugin's code. XCode built the plugin successfully, but when I went to apply it on a layer in AE, it just gave me an error that said "Could not find entry point", then it said "Invalid Filter". So, that was the same error I was receiving on the windows side. Is there anything I'm missing to make sure the .dylib loads correctly if I don't want to do run time loading?
Ah, ok, that should make things easier. So where did you put your dylib file? Same as with Windows, it needs to be in a location referenced in the path environment variable or in the location of the calling application. Easiest thing to test will probably if you put it in /usr/lib or /usr/local/lib (for global access) or ~/lib for the current user only.
See also here: Using Dynamic Libraries