Effects to only affect shadows matte

[Constable_Comfortable]

I'd like to create a hatching animation look by applying a fractal noise effect to the shadows of my Illustrations but I want to avoid having to mask and key everything out by hand. 

Is there a way I can filter by luminance value and apply the effects only to the shadows of an image?

matte

[Mylenium]

Unfortunately not really. Your values are all too same-y, so all the channel magic in the world can't produce a clean matte. You can try to work based off the green channel, but you will to massively have to adjust the levels to clip the shadows or even things out. That would be no different whether you do it with Photoshop's Channels palette, adjustments and calculations or the equivalent tools in AE such as Channel Combiner, Levels, Curves and so on applied to duplicates and pre-comps. The problem really is that your shadows are all mid ranges and you have to find ways to stencil out the darker an brighter areas around them. I'm sure it's doable, it will just require lots of tweaking.

Mylenium