In a properly tuned system, I don't think switching between virtual
desktops in the window manager should have any impact on a realtime
priority Jack. Nothing special happens here but a bit more X11
redrawing activity.
Changing to one of the linux *console* virtual terminals from
X is a different story, that has documented high probability of
overrun.
That's the one I was speaking of, and I think you're right (as always
Jesse!) to make distinction.
Thanks,
Mark