Motion tile works on layer size. If the layer has a mask or transparency then you will get funky edges. Show us a screenshot of your problem with the modified properties of the layer revealed. Select the layer, press UU to reveal them, have the layer asset selected in the Project Panel so we can see the problem, then embed the image so we can see the problem. This is what happens when you have a mask on the layer:
The values in Motion Tile are percentages. If your layer is the same size and the comp and you set the Tile Width and Height to 50 (50%) you will get the full layer scaled to 50% and tiles that fill the frame. If you then set the Output Height and Output Width to 200 (200%) you can scale the layer to 50% and still fill the frame. If there is any transparency from a mask, you will see edges like you can see in the screenshot.