how can I make the audio spectrum less sensitive

What Rick said. The answer is not in AE. You tweak a duplicate of your audio file in an audio processing tool like Audition until it produces "nice" spectra and reference that in the effect from an otherwise muted layer.

Mylenium

The only controls you have on the effect are in the UI. You just have to adjust them. You can also modify the audio track. For a lot of the audio that I use to drive animations or effects, I will create a separate audio track especially modified and equalized to use just to control the effects. After Effects has limited audio effects. You can apply them then pre-compose the audio track and move all attributes, then use the pre-comp as the source for your audio spectrum effect. A much better tool is Adobe Audition. I often modify the audio so much that it is almost unrecognizable, but it gives me the frequencies and the levels I need to drive the animation. That audio is turned off in the timeline and the original audio is used for the project.