"Any idea how to reduce file size without loosing video quality?"
Well, those are basically the two variables in play and they are tied together. The higher the quality, the higher the filesize. So there is no actual way to reduce file size without losing video quality. You just need to find a balance.
The reason why the file size is going up in this situation is that the bitrate on the videos from zoom are absurdly low. It's something like 2-3Mbps. Maybe even less than that. So if you wanted to match that bitrate that's probably about the setting you want to go for, but keep in mind that it's not a straight through conversion where the bitrate gets maintained. The video is being re-encoded (created from scratch, basically) so you are adding a "generation" of loss. So, I would at the very least up the bitrate a little bit from the source to help compensate for that loss. At the end of the day it's a zoom video, so I don't think people are expecting perfection when it comes to compression.
If you want to get a better idea of the bitrate of your source files you can look in the tree view of Media Info. It's a handy app to have around as an editor.