how to remove holes from overlapping shapes

[uudens]

I use AS3's drawing api to draw filled shapes, but when they overlap, instead of filling the overlapped area, it becomes clear again, with no fill.

My question is, how can i get rid of this behavior, i need to just fill everything, is there a way to somehow "flatten" the shapes?

( i can't use bitmapdata on it because i need to use it as a mask, and if i use bitmapdata to make a bitmap of the drawn shapes then the mask is just a rectangle )

[Greg Dove · Accepted solution]

The simple answer is that you can't with the original drawing API.

EDIT: Actually: are you closing the fill and re 'beginning' it between the shapes? If you are looking to generate a 'union' type shape without the overlap of the fill regions or the 'holes' then the original answer stands.

I think you should be able to use the bitmap approach if you set both the mask and the maskee with cacheAsBitmap=true but this will be alpha based masking depending on the alpha values in your drawn shape, as opposed to a clipping mask.

Other approaches (the 'complicated answer')

fp10 lets you do this with the GraphicsPath object and the winding property depending on the direction that the intersecting shapes are drawin in. You could check that out if fp10 is an option for you.

Beyond that, for fp9, you would need to do geometric math and detect intersections and 'union' the data for the drawing path. A considerable effort required here.

[Greg Dove]

Here's a quick example:

Check out the comments

function drawRectangle(dir:uint,topLeft:Point,bottomRight:Point,target:Graphics):void {
    with (target) {
        moveTo(topLeft.x,topLeft.y);
        switch (dir) {
            case 0 :
                lineTo(bottomRight.x,topLeft.y);
                lineTo(bottomRight.x,bottomRight.y);
                lineTo(topLeft.x,bottomRight.y);
                lineTo(topLeft.x,topLeft.y);
                break;
            default :
                lineTo(topLeft.x,bottomRight.y);
                lineTo(bottomRight.x,bottomRight.y);
                lineTo(bottomRight.x,topLeft.y);
                lineTo(topLeft.x,topLeft.y);
                break;
        }
    }
}
var tl:Point=new Point(100,100);
var br:Point=new Point(200,200);

var s:Sprite = new Sprite();
addChild(s);
var g:Graphics=s.graphics;

g.lineStyle(1,0,1);
g.beginFill(0xff0000,.5)
drawRectangle(0,tl,br,g);
//If you include the line below it will overlap the fills, but drawing clockwise or anticlockwise makes no difference with fp9 api and the original "evenOdd" fill rule:
//g.endFill();g.beginFill(0xff0000,.5)
tl.x=150;tl.y=150;
br.x=250;br.y=250;
drawRectangle(1,tl,br,g);

learn more about the winding rules with fp10 here: http://www.senocular.com/flash/tutorials/flash10drawingapi/

[uudens]

Thanks, Greg, Your answer was helpful!

Unfortunately I can't use flash player 10 and I also can't manage the direction of the shape since it's drawn by the user. I guess i will have to come up with my own ghetto method of achieving what I need..

Seems like it would be easier for flash to just fill everything as opposed to calculating which points are "outside" and which points are "inside" for a fill, but I guess that's not how it is.

[Rothrock]

Well part of the problem with the idea that "flash just fills everything" would still be that you need to know what is "inside" and what is "outside." Unless you actually mean "EVERYTHING," but that wouldn't be very pretty, now would it? The method that Flash uses is a very reasonable and mathematically sound method for determining that. Just because there are other ways doesn't mean that the original choice of how to create the drawing API was wrong. And as pointed out, it has since been expanded.

If you are having users draw a shape and then filling it I think perhaps what you want is a convex hull finding algorithm. Oh darn, as I say that I realize that the shapes drawn might not be convex and I haven't seen anything that is easily implemented/understood on how to generate a concave hull.

Of course my curiosity is peaked a bit here. So I'm going to work something up. If you think it might be helpful I'll post it in a little bit.

[Greg Dove]

@uudens, I think the easiest way to do this is to use alpha masking as I said my first post, with the drawn shapes alphas at 1.

Example here, expanded from the previous code:

function drawRectangle(dir:uint,topLeft:Point,bottomRight:Point,target:Graphics):void {
    with (target) {
        moveTo(topLeft.x,topLeft.y);
        switch (dir) {
            case 0 :
                lineTo(bottomRight.x,topLeft.y);
                lineTo(bottomRight.x,bottomRight.y);
                lineTo(topLeft.x,bottomRight.y);
                lineTo(topLeft.x,topLeft.y);
                break;
            default :
                lineTo(topLeft.x,bottomRight.y);
                lineTo(bottomRight.x,bottomRight.y);
                lineTo(bottomRight.x,topLeft.y);
                lineTo(topLeft.x,topLeft.y);
                break;
        }
    }
}
var tl:Point=new Point(100,100);
var br:Point=new Point(200,200);

var s:Sprite = new Sprite();
addChild(s);
var g:Graphics=s.graphics;

g.lineStyle(1,0,1);
g.beginFill(0xff0000,1)
drawRectangle(0,tl,br,g);
//If you include the line below it will overlap the fills, but drawing clockwise or anticlockwise makes no difference with fp9 api and the original "evenOdd" fill rule:
g.endFill();g.beginFill(0xff0000,1)
tl.x=150;tl.y=150;
br.x=250;br.y=250;
drawRectangle(1,tl,br,g);
var maskee:Sprite = new Sprite();
with (maskee.graphics){
    beginFill(0xff00ff,0.5)
    drawCircle(0,0,100)
}
addChild(maskee);
stage.addEventListener(MouseEvent.CLICK,toggleMask)
stage.addEventListener(MouseEvent.MOUSE_MOVE,movehandler)
var mtoggle:Boolean;
function toggleMask(e:Event):void{
    mtoggle=!mtoggle;
    maskee.cacheAsBitmap=mtoggle;
    s.cacheAsBitmap=mtoggle;
    if (mtoggle) maskee.mask=s;
    else maskee.mask=null;
}
function movehandler(e:MouseEvent):void{
    maskee.x=e.currentTarget.mouseX;
    maskee.y=e.currentTarget.mouseY;
    
}

       @Rothrock: Interested to see what you come up with. I think in the general case you do need to evaluate winding for the paths and then perform intersection tests before you union them into a single path. These are obviously easier for lines compared to quad curves. It's a reasonable amount of work, although there are algorithms floating around for all the bits and pieces. I suspect there are a bunch of edge cases. We're working towards adding in support for geometric shape operations like this in the degrafa library which will be available for use in as3 only (non-flex) projects before too long.