Color ID bakes weird sometimes

After importing a model and baking textures where only one color ID color was used I'm getting strange effects like below:

This is all one object where I took multiple objects and combined them to fit on one texture sheet. Importing the model goes well. But after baking I get this strange distortion. Adding layers of materials do not cover up these distortions. If you look closely you can see the blotches end up on other parts as well. I do this because importing the whole model at once is guaranteed to cause this problem. Since I started breaking the model into textures I haven't had this issue but here it is again and I have no idea how to solve this problem. Can anybody help? To clarify the long tube above and everything else in the picture should be just one color.