Answered
Remove normal from previous layer (Edit - Height not normal)
Probably an obvious way to fix this but I couldn't figure it out but I have a layer of wood which has wood grain normal maps how can I erase them so that the place that I have a rivet doesn't have wood grain going through it?
