How to use brush to paint masks

[Dougal5E75]

Hi I'm quite new to after effects so I apologise if this is very basic but I've struggled to find a solution.

I've been using the compositing feature so that the hue effect I've put on my layer can be determined by the shape of a mask but it seems like I can only really change the shape of the mask using the rectangle and pen tool whereas I'd like to be able to paint on the mask with the brush tool and therefore paint on the parts of the layer that have the effect on. Is this possible?

[Omid Pakbin]

Hello!

I could be wrong, but Rotobrush might be what you need. Take a look at this: https://www.youtube.com/watch?v=LaC81sHVu0g

Hope this helps.

[Dougal5E75]

In terms of photoshop talk, it seems like rotobrush is the quick selection tool whereas I want something like the brush tool, which doesn't lock to the outlines of the image. Also, I'm not just looking to separate the image I'd like to create a mask layer for my selection as my end goal is to tie that mask to an effect via compositing so that only certain areas will be affected by the effect. I'm basically looking to get the same effect as an adjustment layer in photoshop where you can brush on the affected area but maybe I'm just asking too much.

Thanks for your help anyway

[Rick Gerard]

I don't have a good tutorial I can point you to for that because it's a bit of an odd workflow. 

Standard masking and rotoscope work uses the pen tool, or Mocha AE to create animated shape layers. Many people animate masks on a shape layer above the layer you want to mask and then use that shape layer as a track matte for the footage. That is my usual rotoscoping workflow and here's a very simple quick tutorial on how to do that I put together a long time ago. 

If you want to use the paint tools you'll have to make a duplicate of the footage layer, open that duplicate in the Layer Panel, use the brush tool to paint in your masks but you will need to select the original footage layer as the source and then select paint on transparent. 

Unless the camera was locked off you'll have to paint on every frame and it's almost impossible to get those kinds of edges to look smooth. 

Almost all rotoscoping work and masking can be improved by combining techniques. I commonly use Mostion Stabilizing along with manual masking as I show in the tutorial above. I also combine procedural mattes using colors, keyint, luminance and other techniques along with some hand roto work. I also often use a stabilized corner pin workflow using Mocha AE, Corner Pin, CC Power Pin, and track mattes to stabilze enlarge and isolate the area I need to work on. 

If we could see your shot or at last a screenshot of your project along with a detailed workflow description it would help us help you figure out an efficient workflow. Please use the Toolbar instead of the drag and drop panel to share images because many users will not download anything just to take a peek.