How to apply expression loopOut to mask

The loopOut() method only works for numerical values or numerical arrays. Mask keyframes contain complex vector information and none of the looping or traditional math functions will work with masks. There are very few options for creating expressions that will work with masks. You can find most of the methods that will work directly on masks in the Expression Language menu in the timeline. 

Unfortunately, there is very little documentation yet on using these methods and you can't just click on one and apply it.

If the first and last mask keyframe are identical you can apply this expression to create a loop:

valueAtTime(time % key(numKeys).time)

It is kind of a sledgehammer solution and it will only loop smoothly only if the first and last keyframe is identical. If they are not identical now you can just copy the first keyframe, move down the line and paste to get identical starting and ending keyframes.

If you are using the mask to mask something then another option is to have the mask on a separate layer, move to the last keyframe, set the out point for the layer at the last mask keyframe (Alt/Option + ]) then pre-compose the layer moving all attributes and apply time remapping to the pre-comp (Alt/Option + Ctrl/Cmnd + t). Then you move to the last time remapping keyframe (k) then back one frame (Ctrl/Cmnd + left arrow), set a new Time Remapping keyframe, delete the original last keyframe, then use this animated shape layer as a track matte for the layer that you originally wanted to mask.

I hope this makes sense.