Can’t get auto reframe to track my subject

I have a minute long horizontal clip that I had to reframe into 9:16 ratio - auto reframe does the job to resize and tracks my subject decently for some parts of the clip, but it will track background people as subjects and pan away from my subject throughout the video instead of moving with it. Does anyone know how to keep my subject centered for the duration of the clip after I've resized, without picking up background people/objects and tracking those instead?