I am working on a project in which I want 5 musicians to walk around a
building and play.  They will carry netbooks (mics and headphones) and
their signals will be sent to a concert hall where it will be
broadcasted.  Additionally, a "conductor" will decide which individual
stream will every one of those 5 musicians listen to.  So I need a
2-way communication on each netbook and 5 in and 5 out on the server.
The server will be wired.
Although some attempts have been made (see for eg. http://www.computer.org/portal/web/csdl/doi/10.1109/MMCS.1999.778626) it looks like currently network (even cabled) latency issues are still too much for performing remotely.
If I understand correctly what you have in mind, and considering the relatively small space-range (a bulding) you might be better using wireless (radio) audio equipment (like radio mics and audio transmitters) which seems to be more established and reliable, although there will always be 'some' latency.
Might Ninjam suite your needs? Long time since I tried it, but when I 
did it worked quite well.
Keep in mind that Ninjam introduces a 'fixed latency' (in musical terms it's usually 1 beat 'out') to try and sync all palyers, which is clever and suits some kind of applications like jamming, but not real-time remote playing.

Kind regards,
Lorenzo