(Flash CS3) Masking using ‘donut’ shapes within movie clips or with AS3 – it’s possible, and not too bad either

You may have tried creating a shape that has a hole in it for a mask in Flash, and it may not have worked. Or maybe it did work and sometimes it didn’t, and you grew some extra gray hairs and threw your laptop at a wall. Or maybe you’re thinking “what the hell is she talking about now?”. I’ll start from (almost) the beginning [if you're not sure about making masks in Flash, read this super old tech support article.. it hasn't changed].

More after the jump.

So a mask can have a shape cut out in the middle, and if you’re working on the Timeline (not with code) and your mask is a shape (not a movie clip), everything is fine. By “shape cut out in the middle” I mean a ‘merge cut’ in Flash: took a shape (say, a copy of what you’re masking), put it over another bigger and different colored shape, and then selected and deleted the smaller one to cut that shape out of the middle of the bigger one.

Select a shape Removing the shape

(click above to enlarge)

This is a common thing to do when creating masks like this, because you can create a “reverse mask” of whatever you’re masking, which is pretty useful. Check out this file and SWF to see what I’m talking about:

Masking that works

(click above to enlarge) File: masking-donuts.fla.

Now say you convert that shape to a movie clip… it looks fine on the Stage, but when you go and publish your SWF you’re in trouble.


(click above to enlarge) File: masking-donuts2.fla

Doh! Yeah, you may have run into this one. Similarly, give your mask the instance name of masker and what you’re masking an instance name of maskee and add this line of AS3 to frame 1 of the timeline:

maskee.mask = masker;

Doesn’t work either. (but hey, AS3 that short and sweet is almost doable eh)

So the player doesn’t support that way of creating a mask for now, but all hope is not lost. There are other ways to create your mask where it *does* work in this situation. In other words, Flash seems to care about the way you make that donut hole in the middle of your mask. And if you do it a certain way, it actually will work regardless of the movie clip.

Let’s check that out.

  1. I go into the FLA (masking-donuts2.fla) and double click my mask that doesn’t work (unlock the layer).
  2. Take the Lasso tool, make sure Polygon mode isn’t turned on, and drag around the edge of my cut out center. You have to be careful to cut around all the edges entirely.


  1. Then Publish.

And now it works. Similarly, remove the masks and add the ActionScript – now it works there too.


Sweet! It’s a clip with a hole. File: masking-donuts3.fla

Whaaat, you may ask? Weird! you may exclaim. Essentially what’s up is that these kinds of masks (with holes in them) created using merge cut or the Eraser tool do not work as a movie clip mask (and hence with code). Made other ways, they tend to work.

So what are the other ways, you ask? Good question. Here are at least some of the other ways you can create shapes that do work as masks:

  • (so what is above…) Deleting the middle of the shape using the Lasso tool. Do note that Polygon mode does not work.
  • Hand drawing your mask with a brush.
  • Drawing shapes around a center area (pretty much same as above – drawing tools). For example, you can draw for squares around an area, leaving the center empty. This will work.
  • Broken apart text, such as the letter “O”.
  • Create an oval with a stroke, remove the center of that oval (just select and delete). Select the stroke and then choose Modify > Shape > Convert Lines to Fills.
  • Paste any shape like a square from Illustrator (or likely another program) as a bitmap, and then make your merge cut in that shape IN FLASH. If the outer edge of your shape doesn’t matter (like you just need a big square with your Flash shape cut out of the middle), this is probably the best workaround.
  • And finally, bringing in a shape from somewhere else (Illustrator). If you have a shape that includes interior “merge cut” shapes in your FLA that you need to be exactly the same, this will give you EXACTLY what you want. I’ll describe in detail below.

So there are three ways you can work between Illustrator (or some other graphicy software) and Flash with a merge cut shape you want to use as a mask. After all, you probably have your artwork in Flash and you need that specific shape you already have as the mask (or you need it to start with and then modify). Which one you use depends on your shape and what you need.

A simple shape: You can copy the shape that doesn’t work for the mask in Flash, paste it in Illustrator, copy it in Illustrator, and paste it back into Flash. However, In *some* cases with Flash CS3, and particularly with curves, the shape may not look the same after pasting into Illustrator. If that’s a problem, then you need to use one of the other workarounds below.

Simple outer shape (such as a square), complex inner cutouts: Paste your square from Flash to AI, then paste it as a bitmap in Flash. Break apart and make your shape merge cutouts using whatever Flash assets you want.

You have a complex shape in Flash:

  1. Export your shape from Flash as a PNG making sure to choose Match Screen so the image isn’t huge.
  2. Place it in Illustrator
  3. Copy and paste it back to Flash
  4. Break it apart
  5. Use the Magic Wand to remove transparent areas. NOTE: It helps after doing this to select the remaining shape and give it a new fill color. You may have some remaining bits to delete. This also makes it much easier to make further modifications to your shape. The freaky thing here (as noted above) is you can actually merge cut the shape at this point… and it will work.

Now you can have a mask with a merge cut hole in the middle that looks exactly the same, and use it as a movie clip and/or with ActionScript.

Refresh on what doesn’t work:

  • Cutting out the middle using shape merge.
  • Cutting out the middle using Eraser tool.
  • Cutting out the middle using Polygon mode of the Lasso.

So is this really a workaround for “real life”? Totally, especially using something like Illustrator (thank you Illustrator and ‘integration’ – although this is just a bitmap so anything like this should work). Even without Illustrator it’s not too bad — just using the Brush tool, I managed to make it work pretty well for a recent animation. Used the brush tool to carefully draw around the shape I wanted cut out in the center (accomplished fairly easily by edit-in-place mode), and smoothed and dragged the inside edges of the shape until it was pretty much what I wanted.

So yeah, it is a little bit more work and since I use Flash, I like to complain about the quirks. But you gotta admit, it is better than nothing at all and not too bad right?

And if you do notice other stuff related to this (or not, doesn’t matter), like some other way better workflow or trick or something else that doesn’t work, please let me/us know in the comments.

Over and out.

Jen deHaan is a consultant specializing in instructional design, web design, WordPress, technical project management, and writing. She is based in the San Francisco Bay Area.

Tagged with: , , , , , , , , , , , ,
Posted in ActionScript 3.0, animation, Flash CS3, instances, masking, masks, symbols, workaround
  • http://blog.debreuil.com Robin Debreuil

    nice post : )

    This is almost certainly because of clockwise vs counterclockwise issues. If you use the lasso tool, it will work if you make the hole going clockwise, but not if you go the other way.

    One other thing you can also do is convert the mask to vector data, and then insure that data is clockwise. Yeah it is a bit extreme, but it should work in all cases. I have an old jsfl file that converts this data (http://debreuil.com/ASDraw/selectionToDrawingAPI.zip) though it is a bit compressed.

    Anyway, mostly just to mention it is probably a ccw issue — and yeah, a Flash bug imo : )

  • http://www.metaspring.com Ann Arbor Web Site Graphics Design

    That is a neat tip. Thanks for sharing it with us.

  • http://www.heromachine.com Jeff Hebert

    Robin’s clockwise tip saved the day. This was driving me absolutely insane, until a fortuitous Google search brought me here, to the great original article and the dead-on comment by Robin.

    Thank you thank you thank you both! I don’t have all that much hair left any more, but what little I have was getting ripped out in frustration.

    And for the record, this is the stupidest bug I can think of. Crazy.

  • http://labs.blitzagency.com/?author=10 Patrick Matte

    Is it possible at all to dynamically draw a donut shape?

  • http://labs.blitzagency.com/?author=10 Patrick Matte

    Oh I just found the answer

  • Fuat Akgüm

    Firstly, thanks for tip.

    I was trying to mask a movie clip with Circle plus another square. But they were in different places in stage so only the upper shape(circle) was masking the mc. Then I have to merge them with a hairline and convert hairline to shape.

  • stahlman

    Thanks this solved my problem. The strange thing is that it previews correctly, but won’t render upon publish to swf.

  • takopus

    Patrick Matte, thank you for your solution. Who knew shapes added with AS3 inside one beginFill session goes in XOR.
    But this do not solve the main problem: mask made of this donut will still be solid circle. Duh!

    But actually there is a solution for this problem too: if you set cacheAsBitmap=true for both mask and masked sprite – it will work perfectly! Regards go to user ‘wvxvw’ on actionscript.org forums.

  • http://www.nickbarfuss.com Nick Barfuss

    You rock! I had this problem, and spent quite a few hours trying to sort it out. Thanks to article I realize I am NOT crazy, and that neither I nor my progammer had errors in our work. The solution I found that worked best in the end is to draw the shape in Adobe Illustrator and then paste it into flash.

  • Mr Galix

    Great explanations, thanks a lot!! :)

  • http://vishwas.gagrani@gmail.com Vishwas

    Thanks, this is one of the weirdest things i have had ever met in my 5 yrs experience in flash. Thanks for this page.
    Btw, got an another similar method. Make the hole, use fill tool to fill an another color. Now drag and remove the filled area away. Delete it afterwards. This too worked and was easy for me! :)