Can't align points to key object (point) unless there are more than two points on the path.

I don't understand this functionality, so I'm hoping it can be explained or fixed.

Sometimes when selecting two or more points, I want to align them vertically or horizontally using one point as the key object. This often works fine as the last selcted point is treated as the key object.

However there are many instances where this doesn't work I can't think of a good reason why. Please see the attached screen recording where I demonstrate what does and doesn't allow align by key object. I can't align by selected either. I'm realizing that once all the points are selected, Illustrator probably treats the selection as a single object with nothing to align to rather than multiple points that can be aligned.

Ideally we should be able to align to any points at anytime just as we do objects. Am I missing something? Thank you.