[LAU] M-Audio fast track pro mic input fails with smaller buffer size

Has anyone else had this problem? For some pieces using live mic input, I wanted to reduce latency by pulling the Jack IO buffer size down to 512 samples from the default 1024. (I also tried 256 samples, but that just made Jack crash.) But with the smaller buffer, the mic gets an additional 30-40 dB of gain, making the input unusable. If I turn the preamp on the fast track down very very low and speak quietly into the mic, the sound comes through without obvious distortion or glitches, so it doesn't seem to be an xrun thing. But it distorts very easily and very badly. If I switch back to 1024 samples, the problem disappears. Why would the driver's buffer size have an impact on the incoming signal's gain? Thanks, hjh