I've created a master Xd "stylesheet" file, and am loading the library from that stylesheet to use in other Xd files. When dragging the component from the "stylesheet" file into its workspace, it works fine. But I've been running into a bug where dragging out a component onto the workspace in another file will create an instance where the bounds are offset from the actual art.
I can r-click and the offset component and "fit bounds to content", which fixes the issue. But If I "reset to master state" it becomes offset again.
Sorry to hear you are having trouble while using XD. Is it possible for you to record a video of your workflow so that I can try to replicate it at my end? You may upload the video to a shared location such as Creative Cloud or Dropbox and share the URL with me. Please share the OS and XD versions of your machine as well.
If it's happening with a specific component, could you please make sure that the component is not corrupted?