I would expect a difficult pano merge especially when a tilt-shift lens has been used. Such shots will not analyse and therefore process conventionally so far as their perspective (the lens axis / subject convergence centre being moved, in effect, off-centre within the captured frame).
Do you get results you like better, from completely standard photos (shot nodally coincident if possible)?
Also, can you please show an example: what particular form of "distorted results" are you noticing, and what would you want to see instead?
Tilt-shift and also pano merging: all arguably a distortion, or (expressed more neutrally) involving "transforms" relative to a natural capture. And those "transforms" happen in pursuit of some goal: it would be helpful to understand what goal that is, for you. After all, some people find converging verticals 'disturbing' in a photo - while for other people, it is the outcome from keystoning correction / lens shift movement that they would experience as 'unnatural'.