It's not file dependent. I have made blank documents with test excel files and it does the same on all of them. I thinkt it's got to be a setting somewhere. I was able to make it work like normal on a mac in the office. Here are some screen shots of what I am dealing with. The first shows me just inserting data merge variables

The second is what happens when I place the cursor between the two variables and hit return. You'll notice the second variable is moved out of the text box as can be seen by the red plus sign in the text box.

The third is to show the paragraph style is the "default style"

The forth shows that that style as no extra line spacing applied.

I've done dozens of datamerge jobs on macs and pcs. Ive never seen anything like this before. Could it be a setting on the computer it's self? This issue only happens with datamerge. When just typing in a text box the return functions normally.
It looks from your screen shots like you have entered a break character, not a paragraph return. If you have an extended keyboard, that will happen if you use the enter key on the keypad instead of the return character on the main keyboard.