It's difficult to mark any one post here as the correct answer to your original questions because there are 2 questions, so let's give it a try:
Firstly, the "correct" answer to the second part - how to get a certain look - was covered beautifully in the third reply here (by R Neil Haugen) and whilst it may look daunting, that is exactly what you need to start doing for grading. Remember too that grading used to be a separate job when authoring houses still actually cared. These days it is far more likely to be some happy herbert somewhere slapping on a stock filter for a "movie look" using presets rather than someone taking the time & effort along with the rather steep learning curve that good grading requires.
It's a superbly written response & should be isolated and made a sticky IMO.
The second part - an answer to your first question - involved what to use if not Encore.
This is far more difficult to answer as there is no single answer. The TMPGEnc tool listed in response 2 (by Jim Simon) will get a rather rough & ready Blu-ray image done, but you can run foul of the rules here extremely easily.
This is because to the best of my knowledge, that tool cannot output a correctly formatted BDCMF replication master. I cannot stress the importance of this strongly enough, as if you create your final product on BD-R then the Blu-ray specifications state categorically that you cannot - repeat cannot sell this as Blu-ray because it is considered by Sony to be what they call "non compliant product" so it must not be sold as Blu-ray, must not use the term "Blu-ray disc" or use any of the Blu-ray logos or trademarks in the advertising copy (online or offline), on the packaging or on the disc itself. If you ignore this and Sony find out they could very well order all copied withdrawn & destroyed.
Why? Because Blu-ray mandates the use of AACS copy protection in all replicas (this is why HD-DVD was killed off, as Sony sold the major film studios on Blu-ray in 2 ways - firstly by promising an unbreakable copy protection system that would end the "scourge of piracy" forever, and secondly by literally bribing the studios with loads of money to sign a Blu-ray only exclusive deal) and not only is this brutally expensive to implement (it literally doubles the replication costs) but the content owner will also require an AACS site license. It used to be the case (and I do not know if it still is or not) that you could work around the site license necessity by hiring an authoring house that owns it's own content producers site license.
So in summary whilst the TMPGEnc tool is fine for home use, using it for commercial titles is a huge risk. This is mentioned in the small print on the main website under "Specifications" where we read the following in note *5:
This application cannot process a Blu-ray (BDAV/BDMV) protected by a Copyright or anti-copy enforcement system.
Which in plain English means "for home use only".
NB - for DVD-Video titles there is no mandated copy protection system, so both ISO masters & folder masters as well as DVD-R masters (I am unsure if a DVD+R master is acceptable given it is also non compliant product) are acceptable by some factories for use as replication masters.
There was also the full spec "DoStudio" tool, originally developed by NetBlender, who were bought out by Sony Creative Software who promptly killed it off as fast as they could before finally killing themselves off a couple of years back, selling some assets to Magix software and discontinuing the rest.
I do not know if Sony's own Blu-Print is still selling new licenses or not.
The only full-spec tools still in development are - oddly enough - the Scenarist tools - with both Scenarist-SD and Scenarist-BD both still in active development and fully supported. Which is seriously ironic, given that we lost Encore because Rovi bought Sonic Solutions out and closed them down - bear in mind that Sonic Solutions owned the rights to the Scenarist platform as well as the authorcore used in Encore, so when Rovi shut them down everything vanished. It ended up with some of the people who used to work for Sonic Solutions creating Scenarist LLC and starting over, including active development - so it is all now available again and I still wonder if the Encore AuthorCore would be license-able again - If I were a gambling man I would place bets on this being the case too, so I believe the field is wide-open for Encore to be kick-started back into life again should anyone at Adobe care enough to try & make it happen, but given the AME Native H.264 Blu-ray codec has been broken for 2 years I am not going to hold my breath.
For regular DVD there are a few options, but the one I would recommend is Media Chance Labs DVD-Lab Pro 2.
Whilst this one is no longer in active development it is still supported and the DVD specs have not changed so this is not an issue. It also has very good support forums, with people who know this tool inside-out on hand for any issues. It is incredibly powerful too, with not only an abstraction layer system that works but full scripting support that can be used either as an adjunct to the abstraction layer, but also a fully independent system allowing you to write your own code from start to finish using a modular system that is very easy to get started with.
It is a truly superb authoring system.
AdChoices