It's because there is no such thing as a physical sort order in the way the files (or the files segments) are stored on your drive. A sort order in any software or operating system is only defined by indexing common files properties like file name or date.
If you want a custom order (like in an audio playlist), the 'custom' order is stored in a separate list. That's exactly what happens with the 'album sort order'. It's a list stored only in the catalog, which does not mean anything if you are not displaying and album and if you have chosen that sort order instead of the sort by date, file name or batch import.