chriswareham(a)chriswareham.demon.co.uk wrote: I discussed these issues at the latest ZKM LAD conference with Paul Davis (jack) and Francois Dechelle (ircam) and the most flexible approach seems to add networking capabilities to jackd and the jack client library. Plus considering that midi over jack is being implemented too you would have both midi and audio over ethernet through jack, available to any jack client without the application needing to be changed. We are interested in allowing clusters of LinuxSampler machines and we here we need both audio and midi over ethernet so I assume devs from the LS team will contribute to the jack networking stuff too. Since we need both midi and audio Paul said it's probably the best to wait that midi over jack is implemented and then start planning how to export this data over network. The cool thing of such an audio/midi network protocol would be that you could write VST/AU plugins on windows/mac which sees the remote machines like they were local plugins. (there is already a commercial audio/midi VST over ethernet solution http://www.fx-max.com/ ) cheers, Benno http://www.linuxsampler.org

Audio traffic has a constant data rate. eg 44.1khz 16bit stereo is 176400 bytes/sec since the audio cards use audio fragments (or periods) of N frames it is natural to send audio using packets over the network of that size (or multiples of it). UDP is the natural choice because of the low latency. Of course there might be a packet loss but I if you go TCP/IP or implement your own retransmission schemes you lose the low latency characteristics. On a local LAN on a non congested network if the hardware is not broken the packet loss ratio is basically zero and AFAIK most of the low latency audio over IP protocols are implemented with this assumption in mind. So I think we should go that route too. You cannot saturate the network till it gets congested because it would not even work with TCP, which would prevent errors but causing so much delays that the audio streams would stutter. Regarding the MIDI data over network: MIDI is event based and in theory there is no upper limit of how much midi events per time unit one might want to send. But in praticle standard MIDI (over a serial 31.250kbit link) is limited to about 3000bytes/sec. This is enough in most of cases but we all know that big MIDI setups need multiple midi interfaces (most professional midi expanders provide 2 MIDI input connectors to achieve better timing in a high track setup and to provide more than 16way multitimbratlity). So far so good. I think one of the simplest approaches would be to send audio packets with embedded midi data over the network. As we know the more information we pack into a signle datapacket and the less handshakes the network devices do the higher the probability that the data flow is reliable and fast. Let's do some math: UDP limits the payload of a packet to about 1500bytes (the ethernet frame). That way it's guaranteed that when we send a block of data it gets delivered atomically without incurring into fragmentation issues (of course if you send it across several routers where the MTU is < 1500 then it's another story but since we are talking LAN ( NIC -> switch -> NIC) fragmentation is not an issue. assume the above stereo stream but we want to transmit floats (jackd uses floats). sizeof(float) (=4) * 44100 * 2 = 352800 bytes/sec assume we send a stereo stream 128 frames per packet 128*2 *sizeof(float) = 1024 bytes , 2.9msec worth of audio data we still have over 400 bytes left for our custom header (eg id remote jack client, jack port number etc) and for midi data. So my proposal would be to limit the number of midi events per audio fragment. let's take simlified MIDI encoding and let's see what we can fit into 400 bytes. Since we want sample accurate midi triggering (which traditional MIDI over serial does not provide) we could do the following: a MIDI command is usally not longer than 3 bytes (let's forget abut sysex etc for now). we could divide 400 bytes into 100 midi events consisting in: 1 byte timestamp relative to the audio fragment (0-127) , this would limit the fragmentsize to max 256 frames 3 byte midi payload This means we could have up to 100 MIDI events (note on/off, controller, programchange etc) per 2.9msec audio fragment ! This is a LOT of events. Just take traditional MIDI over serial: a NOTE-ON takes 1.1msec to transmit (3byte) this means in 2.9msec you can barely send a NOTE-ON and a NOTE-OFF and you consumed almost the full channel capacity. In the case of serial MIDI you achieve a maximum about (3byte MIDI events) 1000 events/sec. In our case we send about 344 packets/sec which multiplied with 100 midi events gives us 34400 events/sec which are SAMPLE ACCURATE ! basically it would be like having the equivalent 34 midi interfaces. Such a (bidirectional) audio/midi stream would consume about 500KByte/sec , which means a 100Mbit LAN could run 10-15 of those at the same time without loosing data. basically it would work as follows: client PC (has an audio card) <----> jack server PC (runs jackd and jack clients like samplers, softsynths, HDR apps etc, no audio card). the client PC would run a jack network client which does the client PC <---> server PC communication the server PC has a special (not implemented yet) jackd input/output driver which recieves/sends the audio data to the network, for the rest the jack clients residing on the server PC don't know anything about the network, to them it looks like a standard jack server. The "clock" to the jackd residing on server PC is given by the client PC. client PC: ----- jack_process_callback() { send_to_network(local_jack_input_port); // non blocking fetch_packet_from_local_queue_and_check_if_new_packet_arrived(local_jack_output_port); // non blocking } ---- server PC: (I'm not yet familiar with the jackd driver API so the naming of the functions will be wrong but you'll hopefully understand what I'm trying to explain). ---- while(1) { receive_audio_input_data_from_network(); // blocking call process() of local jack clients send_audio_output_data_to_network(); // non blocking } ---- on the client PC we send out the data packet in non blocking mode and the next call fetches the next audio fragment from a local queue. Of course during the first iteration the queue does not contain anything so we will prefill it with 1-2 fragments worth of silence. 1 fragment is the minimum needed to work (since the fetch_packet.. will be called only a few usecs later after send_to_network because it's non blocking). We can prefill with more fragments which will of course increase the end to end latency but will at the same help to eliminate network jitter problems. I think with the above approach end to end latencies of < 8-10msec can be achieved which means you can play a cluster of sampler/softsynths live and having rendered the audio in real time on your client box containing the audio card. The audiocard-less PCs will all stay in sync with the PC containing the audio card thanks to the blocking receive_audio...() call. Do you see particular flaws in this proposal ? (of course it would need to be adapted and made more flexible so that it can deal with an arbitrary number of jack audio/midi ports) It's the simplest I can think of and as said above, adding all sorts of error correction, synchronization etc would buy us almost nothing and probably never achieve the same low latency that this system can deliver. Of course we will not know before we actually turn ideas into working code. But if my approach fails then I'll be glad to offer a few beers at next ZKM :) (Steve H. ? :)) ) cheers, Benno Hans Fugal wrote:

On Tue, Jun 22, 2004 at 09:30:19AM -0700, Michael Ost wrote: It doesnt. I'l note that rolling yet another MIDI over UDP/TCP protocol is really stupid, there are 3 that I know of: IEEE-whatever, RTP-MIDI and OSC. VNC will send big packets, so oyu will need some kind of QOS support or you will get dropouts. - Steve

On 22 Jun 2004, Michael Ost wrote: You dont want to compress if you want any kind of predictable latency. No, udp has no sequencing. You'll need to define your own sequencing protocol. You don't have to worry about out-of-sequence though unless you have multipath. On a local LAN with ethernet switch, it should be impossible to get out of sequence packets. VNC *alone* can saturate 100bt single handedly. If you want to do that youll want GbE. Fortunately GbE is dead cheap these days. -Dan

On Tue, Jun 22, 2004 at 12:37:49 +0100, chriswareham(a)chriswareham.demon.co.uk wrote: I'm not familiar with this sytem, but several of ther others like it realy are audio over /ethernet/, not audio over IP, which means you need dedicated (and sometimes custom, eg. in MAGiC) NICs. - Steve