Gradient Maps for matching color tones in different photos

I have two different photos and I want to match the tones from one photo to a different photo. I use the Color Sampler Tool to sample the darks, mids, and lights from one photo, and then I make a standard gradient map adjustment layer and add the values from the other photo to the stops in the gradient map. However, the results do not seem to match.

What am I doing wrong?

Thanks.