Restricting Access to documents when User moves to a new group

How to stop the user from accessing documents he/she routed and signed from another group? We have Enterprise Account but when we move the user to another group, the user can still access the documents from his/her previus group. Thanks