This checkbox has never worked as it should, but now it's more broken than ever.
It's supposed to do two things. In addition to the obvious, embedding the profile, it should also turn on the color managed view, using the system monitor profile to display the file correctly in the chosen document color space.
At the moment it doesn't use the monitor profile. People with standard monitors may not immediately notice this, but with a wide gamut monitor it becomes very obvious (oversaturated). It behaves like Save For Web with Preview set to "Monitor Color".
In the old Save For Web, there were two separate settings - an "embed profile"-box, and a rolldown for "Preview". Setting this rolldown to "use document profile" turned on the color managed view (and this still works reliably). It actually makes perfect sense to roll these two into one, so that's fine. But it doesn't work.
One more thing: Please make the "embed color profile"-box checked by default! Or at least please make it stick once you check it! As it is now, you have to check it every time.
I realize that with most modern browsers, embedding the profile is not strictly necessary. The browser will assign sRGB if it's missing. But these images may be used for other things too. And at 3kB, you can no longer make the argument that an sRGB profile has any significant impact on file sizes. If you need to strip the profile you can still do that - but embedding it maximizes the chances of correct representation of the file.
Plus the added advantage that having a color managed Export image will actually match what you see in Photoshop. This alone should eliminate a heap of user confusion.