XMP files

I want to have the xmp sidecards, but don't want them to slow things down. Is there a way around this?

xmp sidecar files do not slow anything down

I don't use XMP files, but rely almost exclusively on the Lightroom catalog. I shoot almost exclusively raw files. But if I want to provide images to others for inspection or approval I won't provide them with raw files, but will export JPEG copies. In the 10 years or more that I have been using Lightroom I have only had to resort to restoring a backup catalog one time. The key for me has always been to ensure that I have good backups of my working catalog.

When needed, the xmp files need to be with the photos, side by side

If you send your RAW photo to someone to look at, they will need the xmp file to read the edits. THe software they use will read the xmp file IF it supports that and it is in the same folder as the photo

Lightroom Killer Tips | Writing XMP Files in Lightroom (why you shouldn’t and when you should) 

If you need to use the xmp file to recover from a corrupt catalog that was not backed up, they need to be side by side in the same folder.

Photography Concentrate | A Lightroom Tip That Could Save You 

Only two reasons to create the xmp file, Sharing and Disaster recovery.

Note, not everything will come back via recovery using xmp files, I think the History for one does not.