This depends on your needs. But I have been using the Organizer for about 20 years and have imported all of my digital images into my catalog. I currently have over 200k files in my catalog, and I can only think of upsides to importing photos into the Organizer. 20GB of files should not take very long to import. (My photos take up about 1.5 TB.) I can find any file I am looking for within a few seconds.
I suppose the only downside would be the additional time to backup the catalog files using the Organizer's backup utility.
So, I would encourage you to catalog all of your photos, even if you only intend to edit a small number of them.