Need help with Baked Lighting Enviroment Issues

I work with super lowpoly meshes and bake highpoly details onto them. Sometimes, I use baked lighting environments as a base, but I encounter glitches when I turn the roughness down. Theres areas where the lighting is removed or warped. It's super strange, and I've never understood why this happens.

I suspect it might be related to the lowpoly mesh or the shade smooth setting I have applied. There are no marked sharps because when I use them, I get rough edges, which I try to avoid.

Does anyone know how to fix this issue? It's driving me insane! Any advice would be greatly appreciated.