I had a lot of latency problems running Ubuntu 8.10 with a realtime kernel. And nothing changed modifying the limits.conf file and adding myself to the audio group. So I decided to install Musix. I think it is the best distro for audio-linux ever. At first glance it looks rather messy but it works as smooth as a Mercedes-Benz.

I've reached latencies of 3 ms. with no trace of xrun nor clicks in the audio, while in Ubuntu I just could barely get close to 25 ms which was insane. I cannot recommend anything else. Trust me.

Good luck!