Perfect hyphenation/ justified type like top book publishers

I think to some extent you're finding what you're looking for; in sampling a dozen books from commercial publishers I find variation in the hyphenation that about matches what I would expect for any given combination of page size, font and margin size, and overall text density. Works from the best departments of the best publishers have commendably few hyphens, but denser nonfiction works have more, and even have "faults" such as stacked hyphens.

Novels tend to have relatively "soft" typesetting — large and loose — that allows a lot of room for tweaking away hyphens. One I glanced through appears to have no hyphenation at all, but that's with very loose spacing of both text and lines. One hyphen per page or spread is not unusual with a novel from a noted writer and a quality publisher.

Most nonfiction work is set much more densely and does not have the "elasticity" of fiction. Most nonfic I look at, with overall denser page typography, have no excess of hyphenation to my (slightly fussy) eye but do have at least a few on each page, sometimes five or six. The goal is not usually not to try to eliminate hyphens, but to minimize them, watch for bad word breaks, and do everthing you can to avoid 'stacks' — two on successive lines, or three, or more than a few within any paragraph, and always avoid a hyphen at the end of a page, if not column.

And, of course, the vocabulary in use has a huge effect. A novel using mostly simpler words can probably avoid hyphenation altogether, both because the 'soft' text flow has enough elasticity to avoid breaks and because many short words bypass the need altogether. A nonfiction book with a more advanced or academic vocabulary and probably 1.5 to 2 times average word length is going to struggle to keep clean, even typography and avoid hyphenation.

As for "how to do it" I think you have all the tools in front of you. It's a matter of choosing an overall page density that matches the material, then adjusting the hyphenation rules to minimize hyphens (and if possible completely avoid stacks or page-enders). There's no secret or trick there; it's up to you to adjust all the elements until you have the best result you can obtain by optimizing the "rules."

And then you go through paragraph by paragraph and apply a whole range of small tweaks to eliminate as many hyphens, especially "bad" ones, as you can. The two tools here are tracking (by tenths of of a point) applied to as small a section of the text as possible to "pop" a hyphen forward or back; and text horizontal scale, by 1% increments, used in place of or in combination with tracking (again, on as short a segment of text as you can) to move the hyphen. Done within small ranges and with some care, neither tweak will be noticeable to any reader.

There is also the luxury, if you have it, of rewriting short passages to move troublesome words to another location within the line. (As an author/publisher, I do it all the time. But as a publisher, even one with editorial license, you may not be able to .)

So the short answer to "how do quality publishers minimize hyphenation," is all of the above, coupled with acquired skill in using both the automatic and manual adjustments, and most importantly with the time to implement the latter tweaks line by line and paragraph by paragraph.

There is no second method that I know of.