Hi! Im kinda new to Photoshop but I ran twice in that problem. So I was editing text (rasterized) with smudge tool and then tried to gradient it. When I pick Gradient Map and clip it to the text layer (which is rasterized) it changes the color of the text but only to the last color from all of the selected colors for the mix. Sorry for my bad english, I hope someone can help me because it bugs me a lot 😄
The way that Gradient Map works is that the lightest color in the image is mapped to the right most color in the gradient and the darkest color in the image is mapped to the darkest color in the image. Since your text is just one color, gradient map is applying only one color.
A Gradient adjustment layer will be filled with the colors of the gradient. Your text layer should be just below the Gradient adjustment layer so that you can clip the gradient to the text. (Alt or Option click the divider line between the text layer and the Gradient layer.)