How to i make button who save flash animation to jpg?

Here is your source code updated. You must publish your SWF for 11.3 or newer. Take a look UPDATED CODE section

var penSprite:Sprite = new Sprite();

stage.addChild(penSprite);

var mouseDownFlag:Boolean = false;

penSprite.graphics.lineStyle(1,0x000000);

stage.addEventListener(MouseEvent.MOUSE_DOWN, mouseDown);

stage.addEventListener(MouseEvent.MOUSE_MOVE, mouseMove);

stage.addEventListener(MouseEvent.MOUSE_UP,   mouseUp);

function mouseDown(e:MouseEvent):void {

    penSprite.graphics.moveTo(e.localX, e.localY);

    mouseDownFlag = true;

}

function mouseMove(e:MouseEvent):void {

    if (mouseDownFlag) {

        penSprite.graphics.lineTo(e.localX, e.localY);

    }

}

function mouseUp(e:MouseEvent):void {

    mouseDownFlag = false;

}

/*****************************************

UPDATED CODE

*****************************************/

var jpeg_bytes : ByteArray;

stage.addEventListener(KeyboardEvent.KEY_DOWN, save);

function save(e:KeyboardEvent):void{

    if ( e.keyCode == Keyboard.SPACE ) {

        stage.removeEventListener(KeyboardEvent.KEY_DOWN, save);

        var jpg_bmp : BitmapData = new BitmapData(stage.stageWidth,stage.stageHeight);

        jpg_bmp.draw(penSprite);

        var JEO : JPEGEncoderOptions = new JPEGEncoderOptions(88); // 88 it's a quality

        jpeg_bytes = jpg_bmp.encode(jpg_bmp.rect,JEO);

        var fs : FileReference = new FileReference();

        fs.save(jpeg_bytes, "image.jpg");

    }

}

If it's a signature I'd recommend PNG, you don't want to lose quality.

This is more of a server-side question than Flash. Flash needs to communicate with a script on a server that will insert the data. Flash typically does not communicate with a database server directly. Common languages like PHP have their own Image processing functioanlity as well as database connectivity. Your job would be to convert the bitmap to binary, send it to your script, have the script encode the data in a format (PNG) and then insert it into a binary-friendly server like a BLOB columns in a database.

you can use the jpegencoder class to save your bitmap.  google for a tutorial if you need help using it.