How to fine tune color conversion between RGB and CMYK (watercolor illustration)

I agree that we really need to see at least part of the original and ideally know which CMYK profile is the target.

However as a general rule, forget about matching the original scan, end consumers will not see or know the RGB colours, just make the CMYK as pleasing as you can. A good place to start is with curves and selective colour adjustments to help “purify” key hues (perhaps with use of channel mixer or apply image to move detail from the gray component into the primary channels if too much detail is lost when cleaning).

Are you absolutely sure they want you deliver CMYK at all? What are their specifications in detail?

Yep. You need to ask the printer which specific CMYK profile applies to their specific process. This isn't generic. The Photoshop default, US Web Coated (SWOP), is not used anywhere outside North and South America.

The Image > Mode command is fine if you're aware of this and know what you're doing, but if not, it's very misleading and potentially damaging.

You also need to know that color spaces don't overlap entirely, and RGB and CMYK spaces overlap the least. The non-overlapping areas are known as "out of gamut", and there's nothing you can do about that except compensate as best you can. Out of gamut colors simply cannot be reproduced in that color space.

Please state the specific Color Spaces instead of just saying »RGB« and »CMYK«.

Also please post the image in question or at least screenshots so people may better understand what you are trying to describe.

What are Photoshop’s Edit > Color Settings?

When using »Convert to Profile« which exact and complete settings have you tried yet? (As for Intents only »Relative Colorimetric« and »Perceptual« would seem to make sense here.)