My standard way to work is to have my pattern Swatch libraries open in various Panels, and I just click on the one I need when I need it. But Illustrator then adds that Swatch to the document's Swatch panel, which can make the document larger than it needs to be and slow down saving the file.
Is there any way to turn this off? I don't need any Swatches being auto-added to Panel.
You'll need to set up a New Doument Profile without the standard libraries and use that when you start a new document. Just start a new document, select Delete Unused Swatches from the Swatch panel flyout and save the file in your User folder.