Two tracks of the same conversation becoming unsynced (in no consistent way)?

Recording a conversation — my side locally into Audition (on Yeti mic), my partner's locally through Zoom. 

I use my track of the Zoom recording only to find where to line up and sync with my local recording, and then use the local track for the actual mix.  

Crazy part is — and this didn't use to happen — the track is lined up in one place and then becomes unaligned immediately, sometimes early, sometimes late.  It's exactly the same length but it's as if the pauses are stretched in certain places, or the conversation is stretched half a percent, or... I have no idea.  It's making me question the nature of time. 

Image attached shows how the (local) track is synced to start, early at the cursor, late at the playhead.  What's going on???