Firefly doesn't know difference between a judge and a graduate.

Adding a negative prompt like "no hat" in your main prompt area will not work the way you want. Adobe implemented negative prompt functionality through an Exclude from image setting. However, with the latest Firefly update, the 'Exclude from image' is not available at this time. Hopefully, it will be back soon and should work as in the link below.
https://www.adobe.com/products/firefly/discover/ai-art-negative-prompts.html#how-to-use-negative-prompts-in-firefly

It generally does not understand "no [something]" or "without [something]".  Try using "hatless" rather than "no hat".