There is a tween library built into AS3, you can read about it here:
Tween - Adobe ActionScript® 3 (AS3 Flash) API Reference
The first part of the problem is knowing that the button was clicked. You add a listener to the button that when it's clicked it than calls a function, like this:
button1.addEventListener(MouseEvent.CLICK,shrinkcircle);
The 'shrinkcircle' function will be what tells the circle to scale, and using the tween library the whole code would be:
import fl.transitions.Tween;
import fl.transitions.easing.*;
button1.addEventListener(MouseEvent.CLICK, shrinkcircle);
function shrinkcircle(e: MouseEvent) {
new Tween(circle1, "scaleX", Elastic.easeOut, 1, .5, 3, true)
new Tween(circle1, "scaleY", Elastic.easeOut, 1, .5, 3, true);
}
The Elastic.easeOut part is the easing setting, there are simpler ones, like this is no easing at all:
new Tween(circle1, "scaleX", None.easeNone, 1, .5, 1, true)
new Tween(circle1, "scaleY", None.easeNone, 1, .5, 1, true);
The 3 in the first example means the tween will take 3 seconds. The second example is set to 1 second.
You will read online that many people use other tween libraries, that amongst other things let you tween two properties under one tween. It wouldn't look any different, and if you do use those libraries you'll have to include their libraries. The one I showed is built in.