[LAU] Building Audio from a Server Core?

[Grekim Jennings]

How difficult or even possible is it to start with a server core system 
and make it useable for a audio?  I don't need a GUI/desktop for this.  
I installed Lubuntu server core to a usb flash drive.  I added myself to 
the audio group, but my usb Audiobox does not seem to be working, 
although it is recognised with aplay -l.  Should it work without any 
additional steps?
I suppose I could build an Arch system, but I was pleasantly surprised 
by the performance of Ubuntu 12.04 (desktop) with the particular 
software I want to run and thought I would try an even more basic 
setup.  Thanks!