Displacement map confusion.

I have been trying to use a displacement map and the result is that the entire layer is shifted by the number of pixels specified when using the filter. I have had this issue before and just gave up out of frustration. I have a layer that is a textured skin layer. There are plenty of dark and light contrasts. I  have opened the layer in a new document, converted to gray scale, enhanced contrast and saved as a PSD. I am trying to put some wrinkles on the skin and want for them to take the contours of the skin contrast. I created a layer with the prospective wrinkles (not very good yet. That is a work in progress). When I apply the displacement map, all that happens is the entire wrinkle layer is shifted by the designated number of pixels in one direction. There is no change of the pattern withing the layer. I hope that this makes sense and someone can tell me the likely simple solution to this.

I have attached the PSD displacement map, the face that I am trying to put the wrinkles on, the face with the wrinkles pre and post dispacement and the steps of the displacement process. You will see in the post displacement that the entire darker portion, the wrinkles, has been shifted inside the highlights, but there is no distortion in the actual wrinkles. I had to do the wrinkles and the highlights seperately as one is a linear dodge layer and the other a linear burn layer. It works out well, because it clearly illustrates my issue.

If anyone can make heads or tails of this, I will be eternally grateful.