Overlapping Opacity with Symbols

I'm working on a project and I have some buidlings casting shadows with trees also in those shadows. I have made my trees into a Symbol object so that I can easily edit one and adjust all the rest at one. However, in this situation, the shadows that are overlapping from the building and the tree are double darkening. I tried playing around with the blend modes but nothing seemed to even it out. I saw on this post to use the Shape Builder tool, however, It doesn't seem to work with symbols. If I can avoid it, I would prefer to not "Break Link to Symbol " and manually do it. It works fine, and gets me the results I'm looking for, but to keep the file as light as possible and simple to modify, it's not ideal.

Another approach I saw was to just use solid colors with no opacity. This works for most areas except where there are many layers in front of and behind others. The example screenshot I have here has lines in front of and behind shadows, so if I wanted to break it up into layers I would need about  4 or 5 layers to manage these items so that they overlap as intended, which is kinda messy as I'd like to keep my objects like trees all on the same layer, not split the components of the tree onto multiple.

Any help is great!

Result after removing the link and pathfinder uniting the shadows: