I have worded things wrong, or not used enough words. I have not found a badly behaved audio program. When using multiple workspaces, the pager used to get from one workspace to another can get swapped out with a low enough swappiness. Clicking on the pager means the pager has to be moved into memory. The pager itself is of low priority and does not disturb the audio, but the kernel code (at least a few versions back when I was testing this stuff) that moves the pager back into memory has a higher priority and can stop audio on the lowlatency kernel.