Animated button scripting issues

Using createjs.Tween, as ClayUUID suggested, the code can be made much shorter and the animation will be eased:

stage.enableMouseOver(); // Should be placed on the main timeline instead
var button = this;
var transitionTime = 0.15;
button.addEventListener("mouseover", Over);
button.addEventListener("mouseout", Out);
function Over () {
    createjs.Tween.get(button, {override:true}).to({scaleX:1.25, scaleY:1.25}, 1000 * transitionTime, createjs.Ease.quadOut);
}
function Out () {
    createjs.Tween.get(button, {override:true}).to({scaleX:1, scaleY:1}, 1000 * transitionTime, createjs.Ease.quadOut);
}

I tried your code, and yeah, it's pretty glitchy. While it should be possible to make it work more stable, I think it would be easier to get stable results by animating it through code.

This will work if you place it inside a movieclip:

stage.enableMouseOver(); // Should be placed on the main timeline instead
var button = this;
var transitionTime = 0.15;
var transitionAmount = 0;
var transitionSpeed = (1 / transitionTime) / 30; // 30 refers to the frame rate
button.addEventListener("mouseover", Over);
button.addEventListener("mouseout", Out);
function Over () {
     createjs.Ticker.removeEventListener("tick", OutTransition); // Remove the out transition
     createjs.Ticker.addEventListener("tick", OverTransition); // Start the over transition
}
function Out () {
     createjs.Ticker.removeEventListener("tick", OverTransition); // Remove the over transition
     createjs.Ticker.addEventListener("tick", OutTransition); // Start the out transition
}
function OverTransition () {
     transitionAmount += transitionSpeed;
     if (transitionAmount >= 1) {
          transitionAmount = 1;
          createjs.Ticker.removeEventListener("tick", OverTransition);
     }
     
     Animate();
}
function OutTransition () {
     transitionAmount -= transitionSpeed;
     if (transitionAmount <= 0) {
          transitionAmount = 0;
          createjs.Ticker.removeEventListener("tick", OutTransition);
     }
     Animate();
}
function Animate () {
     // This is where you update the buttons state
     button.scaleX = 1 + transitionAmount * 0.25;
     button.scaleY = 1 + transitionAmount * 0.25;
}

While this is a lot more code, the benefit of this is that you can get identical animation out of multiple different symbols, without creating identical animations for them. Also code like this could be added to a class, so that you can turn any symbol into an animated button, with only one line of code. Then you would only need to update the code in the class, to update the behavior of each of the buttons you're using, so you wouldn't need to change the code in every single button.

If you want me to explain how to do that, just let me know.