I'll double-check. Plugging and unplugging with
snd-usb-audio works
great most of the time.
If you do not need hotplug for other devices, you could completely
remove them from your boot process and load snd_usb_audio statically
during boot time in one of the boot scripts. That's how I do it,
because it also allows to pass parameters to the module.
Best regards
ce