I used to be of the same opinion. Currently running a 3.6.11-rt kernel, and getting very low latencies from it, on a modest 5 year old laptop.
We're talking 0.666 ms, while compiling a kernel, 0 xruns.

Notes:
-logged in as root (yes yes, i know)
-rtirq running on startup to fix interrupts
-wifi is a killer, turn it off (Thanks to Jeremy's LAC workshop on RT-ing a system, I didn't know about it till then)
-cpu-scaling can cause xruns, cpupower frequency-set -g performance to run all cores at 100% all the time.
-i kicked out bluetooth to be sure
-as few daemons as possible
-there is no proprietary driver for the Ati HD2400XT integrated for the 3.x series kernel. Hence "normal" graphics, no 3d acceleration

Details:
Acer travelmate laptop: Intel Core2Duo 2.0 Ghz
Arch linux based system: Linux audio 3.6.11-rt25-1-rt #1 SMP PREEMPT RT Tue Dec 25 00:18:17 CET 2012 x86_64 GNU/Linux
Soundcards: RME Multiface II, running trough PCMCIA adapter. Also have an Echo Indigo DJ card, PCMCIA too.