Rod, the LMS vendor's support guy was very clear that the LMS does not control the size of the new window. You're right that they know nothing about Captivate, and they don't really care. We remoted in to the client's system yesterday and couldn't find any relevant settings.
Something is controlling the size of the window, though, because when the next course opens, the window resets to the size of the Captivate file (i.e., without room for the TOC or playbar). It doesn't retain the size of the previous window.
Erik, we can't try it with a different browser, and if we could, it wouldn't help. The client is not going to change browsers so their users don't have to maximize their screens. 
I looked at the published files, and couldn't find any settings. There are no longer any .js files in the published output - just xml stuff.
Thanks for the feedback, both.
There should be a 'SCORM_utilities.js' file in the root of the published project.
Within that file should be lines similar to these:
| COURSE_WIDTH | | : "641", |
COURSE_HEIGHT | | : "511", |
Those numbers are, of course, specific to a published course that I opened here to copy/paste from.
Rod's reply prompted me to check again and he's right, those numbers don't affect the size of the window. They do affect the size of the actual presentation within the window, but not the size of the browser window itself.
But there has to be some way for the LMS to 'understand' the native size of the presentation - I know I've seen the containing lesson window scale appropriately on launch...with the lesson properties set to use the project's native size.
At any rate, the browser opening to the size of the previous window is default behavior (try it; you can resize an IE window small, close it, then reopen it and it'll reopen at that same size). So I'd guess your LMS isn't passing anything at all to that content browser window, so it just goes to the default.