How the Color Management works in InDesign?

There's no simple "conversion code", so if you're looking for a simple mathematical equation, there is none. I suggest you truly need to study Color Management. You can find lots of info in the Internet about the subject. Start there... as much as we're here to help, we are not here to provide you an entire free education. But here goes...

Color Management can be done in many places, it's not just in InDesign or whatever particular app you are using. It can also be at the System level. e.g a Mac has ColorSync.  No matter which is used though, they are all very complex software infrastructure that utilise ICC Profiles to convert colors from one device to another, that when properly installed and selected, can help maintain color consistency.

Back to its simplest concepts though, they are all using the set of data in each ICC Profile as a "look-up table" that have been carefully measured using a very accurate device called a spectrophotometer. e.g. an offset press will print out a very detailed sheet of hundreds of squares of colors on a particular type of paper, and these are all measured and stored in a file for that specific press for that specific paper: its ICC Profile.

Your monitor, when it was manufactured, will also be similarly measured and all the colors it can display are carefully recorded and stored in a specific ICC Profile for it, and is usually installed by your computer when you connect the monitor. Yes, there will be adjustments you will need to do, some are automatic some are manual, e.g. say if you turn the brightness down on your monitor, that chnages how its going to be displayed, so most computer systems have simple calibration utilities to make a revised ICC profiles for your monitor's current state. (High end professionals will employ a calibration device/spectrophotometer to measure their monitor even more accurately).

So to answer this one SPECIFIC question... "If i apply RGB color to rectangle shape in InDesign, how the final output color will be display in the monitor?

InDesign has its Color Settings (under the Edit menu) and what you have set there determines how colors are handled. In many cases the default for RGB in these settings is to a standard sRGB ICC, which is roughly equivalent to most monitors, but never exact. So, if you draw a box with pure Red in RGB (that would be 100R 0G 0B), InDesign passes that value to your computer system, which will then send it to your monitor and will "look-up" in the Monitor's specific ICC profile what values it needs to use to display that Red accurately to YOU. By the time it gets to the monitor it may be that it needs to uses the values 98R 3G 4B to match the Red in your document instead of 100R 0G 0B.

This in NO way changes the Red value in your Document; This is only for display purposes.

However, if you were to PRINT this Document, say to a typical CMYK-based press, InDesign will look up the printer's specific ICC profile (in ID's Color settings you might see a CMYK setting of e.g "SWOP Offset Coated" for an old-school offset press) and finds what it needs to send to the printer in CMYK to match your red as closely as possible. It may find that the closest match to your Red is 2C 92M 99Y 2K. While this "2C 92M 99Y 2K" will look fine on a standard offset press on the right paper. it will look completely wrong if sent to a completely different printer and paper, say an inkjet printer on gloss photo paper, you would need to make sure you have selected the right Profile for your intended output. 

What this all comes down to, is why exactly are you asking for a specfic "conversion code"?

We've already repeatedly explained the concepts to you, but you still continue to ask the same unanswerable question. It would help to know what your end goal here is.