Normal map only works on one side of object

I am working on a packaging project and I want to have this embossed flower on the front side of my bottles. I am trying to do this with a normal map in Dimension, instead of adding it in Photoshop after rendering. This worked fine on one object, but with the rest of them it only wants to apply it on the corners for some reason. When I adjust X offset it just dissapears under the front side. I tried removing my graphic and generating UVs again, and it's still doing it. Also tried detaching material styles on that segment of the object.