How do I keep a character from Synching to mp3 voice file?

I'm a newbie to Character Animator and I've succeeded in getting puppet to sync to an imported voice file on mp3.  But I want to use some mp3 voices that the character does not speak.  Yet when I import ANY voice file, the character speaks it, even if I have not clicked on the option to sync the characters lips to the voice.  How do I prevent the puppet from moving its lips to an mp3 voice?