Hi Paul, thanks for asking — a caption/subtitle track isn't a normal clip track, so it can't be linked or grouped to the audio, and moving the audio alone won't carry it with you.
To shift them by the same amount: first move the audio and note its new start (a marker or the playhead placed at its start helps). Then click the first caption segment in the Timeline, Shift-click the last segment you want to move, and drag any selected segment. All the selected captions move together, keeping their relative timing, so you can line the first one up with the audio start.
Turning on Snap (S) can help you align to the playhead or marker. Hope that works for you!