I don’t think there’s any way to disable the Enter shortcut that selects the first field. It seems to be in the set of keyboard shortcuts and gestures in Photoshop that are hard-coded.
What action exactly is mapped to your “unselect” shortcut? Is it:
- OK, or the [√] Apply button on the options bar, or the Enter key
- Cancel or ([x] button), or the Esc key
- Select > Deselect (a marquee selection)
- Select > Deselect All Layers
This may be important in case there is a better shortcut for the particular way you are “unselecting.” If there is a better or more appropriate shortcut, you might be able to avoid conflicting with the long-established Enter key shortcut.
For example, you say you map it to the Enter key, but in Photoshop that does not map to any of the actual deselect shortcuts. The Enter key is a shortcut for OK/Apply. Deselecting is only a side effect of applying the pending edit, so using the Enter key also means you want to apply any pending changes. If you want a pure deselect that does not change the state, mapping to a different shortcut might be more appropriate.