This is all too common.
I have honestly lost count of the times I have heard stories of changing from one OS to another one (it might still be called OS-X but in reality each revision is a different OS using the same name to maintain the myth of "Apple superiority"). So many people I know who run Logic Audio & PT on the same system get this problem. They change the OS and then half their software simply will not run any more, plugins fail or else load in one app but not the other, causing vast amounts of time either online to support or on the phone to support trying to get things working again yet they usually blame everyone but Apple & themselves.
An OS change is a major thing, and a very big deal indeed - before doing such a drastic thing it is a really good idea to make saure everything is properly compatible as well as making sure all various drivers are also compatible too (sound cards, graphics cards etc)
Encore will never be supported on Mavericks simply because it is End Of Life - Sonic Solutions (who licensed the Encore AuthorCore to Adobe) were swallowed up by Rovi (whose business model is based on streaming on demand) who killed off Sonic Solutions at the end of 2012, finally withdrawing all support from December 31 2013. This affecvted Scenarist SD, Scenarist BD 4 & BD5, Total Code, CineVision, all of it - gone. The argument was that "optical media is dead" but my cynical mind tells me they saw an opportunity to take their main competitor out of the game & went for it. Companies who had spent tens of thousands of pounds on seriously expensive high end authoring & encoding tools suddenly found themselves on their own with no support. Nice move, Rovi.
Another victim here was Adobe, who found themselves with no way to relicense the authorcore so bye-bye Encore. Adobe did *not* stop supporting Encore - the people who bought Sonic Solutions killed it off, and I do not see Adobe starting from scratch & building their own tool to do this as the development costs would be enormous.
Alternatives?
On a Mac, very few - if any. Off the top of my head I can think of Sony Creative Software for PC (Vegas Pro 12 will do written discs but not replicable ones) and DVD-Logic (Not too sure what OS are supported - definitely PC, but not at all sure about Mac). If you need to get replicated discs then AACS is a must and you have options of DoStudio or else Blu-Print (Rivergate software created an add-on for Encore to allow proper BDCMF formatting but this also will most likely not run on latest Mac OS).
In short, Sony are now the only player still actively in the game and they do not run on Mac.