What is best option for high quality(encoding) video ie MP4 or some other format?

There are multiple things that affect video quality.

• The resolution of your original artwork. Are you using Photoshop or Illustrator? What resolution is your Photoshop file? Have you set the artwork to hi res in illustrator under Document Properties? (I personally use Illustrator with a 6000pt square artboard, with doc properties set to high res - then I scale down inside Character Animator. I don't know which settings matter here - just sharing what I use with good success. I use large artwork so I can zoom right in on the eyes etc without losing resolution.)
• The resolution of the video you created. This is in the scene properties. I use 1920x1080 (which is 1080p).
• The Frames per Second affects the smoothness of the video. 24fps will be smoother, 12fps was the old default (which still looks pretty good).
• Someone recently had a problem in the forum where they had changed the scene window scale factor, then did not realize all their recording was at the wrong scale. You might check that the scale factor at the bottom of the scene window is set to "fit" so you are seeing the full screen as will be in the final video (my screen below has 49% - the right value "fit" computed to make it fit on my screen)

Screenshots can help a lot in understanding more specifics of your problem. E.g. does the animation look good inside Character Animator, but bad when you generated the MP4 file? etc.  What is the framesize you picked? What where the Adobe Media Encoder settings you used? Anything you can provide can help. There are lots of places you can adjust things - which means lots of places things can go wrong!

Oh, here are my Adobe Media Encoding settings - which I think is the default. I assume "Match Source" means media encoding picks up the values from Character Animator. If you pick different values, I assume Media Encoder may scale the image, causing blur.