Panorama - align panes to reference at edge that remains unaltered

To elaborate on the subject title, I am trying to manually stitch 6 panes together which all have a lot of overlap (I could even get away with less). Shot at 24mm full frame, no change in focus or aperture. I have a building on the left side that I do not want to be distorted, the right side of the Pano is trees so it doesn't matter as much about distortion. 

When using the various photomerge options it alway results in the building being distorted. I have tried locking (using the 'lock all' button in the layers panel) the left pane but it still gets distorted. I think that perspective merge may be the way to go but I get a full scratch disks error each time I try to align. When I get chance I'll try reducing the images to small JPGs to see whether perspective works. Am I going about this the correct way?