To build on Kimaldis' reply, the custom order of photos in a collection created from the SDK is the order in which photos are added to the collection. If you want to change the custom sort order of an existing collection, you'll have to delete all the photos from the collection and re-add them in the desired custom order. (That can be problematic with very large collections, since LR uses an inappropriate O(n^2) algorithm for deleting from a collection.)
The SDK doesn't allow a plugin to change the current ordering of the Library grid and the filmstrip, e.g. from View > Sort > Capture Time to View > Sort > File Name. So if your plugin wants to present a custom order to the user, it should display a message with the do-not-display-again option, e.g. "Remember to do View > Sort > Custom Order".
[Use the reply button under the first post to ensure replies sort properly.]