Second person in adobe sign custom workflow cannot not decline signature

I have created a custom workflow to approve invoices. The first round goes to our compliance team as an approver and as a group recepient only requiring one of them to initial and approve. Next it routes to our project manager for review and signature. The approver has the ability to decline to sign and provide a reason for declining but everytime the project manager tries to decline to sign it gives them the error in the attached picture. I have done the same thing with a test workflow where I am the approver first and then  the signer second and I run into the same issue. I can decline it as the approver but after that step and I am the signer, it gives me that error when trying to decline it.

Is this because another step in front has already approved it? Any work arounds to allow somebody at any step of the process to decline it even if it has been approved ahead of them? Is there a button I am missing when setting up the workflow? Appreciate any help!