There is a lot of processing going on, not to mention drive
activity and memory allocations. If you don't have a big system
then sending large quantities of email via CFMAIL will have an
effect until it is done.
We had a similar issue years ago (we run 32,000+ mailers
daily) and decided it was just simpler to build a second small
server to just run the mailers each day so it did not affect the
primary server. Unless you want to build a honking massive
powerhouse, that is likely the answer that will best suit you too.