Is there any advantage in using a 64 bit distro for audio? Do the
advantages outweigh the difficulties?

For what I do, live MIDI synthesis, the advantages do outweigh the difficulties.  First of all, when I compile application  source, I compile -march=native -mtune=native.  This gives significant performance increases.  I don't know what would happen if I tried that on a 32-bit OS running on 64-bit hardware, but even if the compilation worked, the result will not use the full CPU capability.  And second, recompiling or not, memory management in general is much more efficient, because larger chunks can be moved in single operations.
-- 
Jonathan E. Brickman
http://linuxsynth.ponderworthy.com