AEGPSetLayerTransferMode not setting a track matte

Has anyone had any luck with AEGPSetLayerTransferMode to set a layer to be a trackmatte of the layer above? It only works for me if the layer is already a trackmatte but it won't ever set it otherwise.

For example if my layer is a trackmatte of the layer above, I can use the function to change the trackmatte type to for example luma inverted. But if my layer is not a trackmatte, the code appears to do nothing. I can also use the functions flags to set a layer to be set to preserve transparency. 

I understand there's a new method in ae 23 and newer which works well, but I'm trying to get backwards compatibility. Here's my code that is not behaving as expected:

AEGP_LayerTransferMode mode = {};
mode.mode = PF_Xfer_IN_FRONT;
mode.track_matte = AEGP_TrackMatte_ALPHA;
mode.flags = 0;
ERR(suites.LayerSuite7()->AEGP_SetLayerTransferMode(layerH, &mode));