[linux-audio-dev] XAP spec - early scribbles

[David Olofson]

On Friday 28 February 2003 09.20, torbenh at gmx.de wrote:
[...]
> random latency ? how do you mean that ?

Latency depends on how you happen to construct the net (order of 
instantiation, connections etc) and/or the actual layout of the net, 
in "non-obvious" ways.

When designing a net, it should be clear in which order plugins will 
execute, or rather, where block latency will be introduced. Normally, 
there will be no block latency at all in a net without feedback 
loops.


> see current implementation...
[...]
> one advantage is with silent sub nets....

I'm not sure it's that easy. What about plugins with tails and/or 
internal state? (Delay, reverbs, most filters, ...) You can't just 
stopp running these when they get no input, or when you don't need 
their output.


> in the current galan implementation a call
> to gen_read_realtime_input( blabla )
> will immediately return FALSE if the input is connected
> to a gain set to zero (well the gain returns false and should not
> traverse its inputs further..)

Sure, but this can be handled without "calling for the data".

Audiality insert FX get NULL input buffer pointers for silence, and 
switch to a different state when they generate silent output. This is 
sufficient for simple "stereo in, stereo out" plugins like these 
inserts, but for the mixer, I'm using a different approach: Each 
buffer has an "in use" flag that is set whenever the buffer is 
non-silent.


> also in some complex event processing nets
> in XAP every plugin gets process call normally only realizing:
>   no input events -> nothing todo

Yeah, but there's no way around that, as you can't know when a plugin 
needs to do some work - whether or not it has pending input events.


> in galan a plugin can register itself to get realtime callbacks...
> (like XAP process callback)
>
> But it does not need to... i will have to do some profiling
> on galan if i have the time...

I don't think one call per plugin for each block matters much. That's 
pretty much all calling there is in XAP. No per-audio-buffer or 
per-event calls in any direction.


> these calls are not ordered ATM but this could be fixed of course.
>
> when migrating away from the global eventqueue to localized ones
> it should be possible to do local buffer splitting in the
> gen_read_realtime_input()

Yes, and that's the main point with timestamped events from a 
performance POV. You can have sample accurate timing without 
constantly impacting the whole net, or even single plugins. The cost 
is only there when you actually deliver multiple events per block, 
and even then, it's a very low cost for most plugins.


[...]
> >
> > > > This suggests to me that "look-ahead" is a special case
> > > > interface for some specific cases; not an idea to build a
> > > > real time API upon.
> > >
> > > yes...
> > > a sequencer could have a clock and a prepare-clock
> > > and then outs and prepare-outs....
> > > this fits on top of XAP.
> >
> > Sure, but I can't quite see where it would be useful. It's just a
> > means of squeezing inherently non-real time features into a
> > strict real time system. Real time controls should never need
> > this, because that would make it impossible no use the plugins in
> > *true* real time systems.
>
> ok... but i see realtime as a subset of offline audio processing...

It is not. Real time applications have some very strict requirements 
that no other applications have. Sinci there is no way around this 
fact, we just have to deal with it, and design the rest of the system 
around the requirements.


> there are some things you must not do in a realtime system.
> but you can do these in a nonRT system...

Exactly - and that is why you must design in a way that allows RT 
hosts and plugins to operate without non RT safe actions.


> you can build realtime systems with C although you can also do
> while(1) ;
>
> i see galan as a programming language.
> so i dont want to lock the user in a system where he can only
> build realtime systems. galan should warn the user if he built
> something not realtime safe though... but it should not be an
> error....

Well, it's not really an error to implement a XAP plugin that won't 
run in real time and/or is nondeterministic. However, I don't quite 
see what you could do in an RT capable *host* that inherently cannot 
run in RT. Multi-pass subnets with conditional loops...? Can you give 
some examples?

I realize that non-linear time processing falls into this category, 
but I don't see how that is relevant to a real time plugin API. I can 
see how it would be *possible* to support it in a plugin API that 
also does real time processing, but I think designing something like 
that is asking for serious trouble. There are just too many conflicts 
between these two ways of working.


[...]
> > I'm not sure... Is this basically about splitting actions up into
> > two events; one "this is what we'll do" event, and one "do it
> > now" event?
>
> yes sort of...
> the idea is to put plugins doing the prepare stuff in front of
> the delay so they could safely fire a worker thread...
> when the worker thread finishes... the delay adjusts
> the timestamp compensating for the non determinism of
> the worker thread...
>
> i see this as a method of trying to guarantee some time
> the worker thread has...
>
> but if the event is late after delaying it, the delay is not
> calibrated correctly...
> but it is not so bad because realtime processing was not
> interrupted...

Right - but then, why mess with this on the API level, and why use a 
delay at all? Well, I can see the jitter reduction advantage, but I 
don't think it's worth it. Just design plugins so that you can tell 
them what you're going to do first, and then have guaranteed RT 
response.

For example, the delay could have a "max delay time" control for 
buffer allocation, and a "delay time" control for setting the actual 
delay. That way, you can use the same plugin for ms delays as well as 
delays of several seconds, without hard-coding the plugin to some 
arbitrary huge buffer size.


[...]
> > > in XAP this event feedback would already be illegal.
> >
> > No, feedback isn't illegal in any way. It's just "late events"
> > that are illegal; only events for this or future blocks must even
> > be found in an input event queue.
>
> why ?
> the semantic of a "late event" is process me as fast as you can
> i am late.

Sure, but timestamps wrap, and as a results, they wrap into the future 
if they arrive late. This can be "fixed", but it requires that we 
agree on a specific maximum allowed time an event may be late.

More importantly though; it's not possible to handle control ramping 
properly, unless timestamps are respected as part of the data. 
"Process ASAP" results in quantization, and that will screw up chains 
of ramps. (XAP ramps are just "aim point" commands, and nothing is 
guaranteed after an aim point is passed. Plugins are not required to 
stop ramping automatically, but can expect to receive a new event in 
time.)


> plugins could report that they received a late event
> to the host so it could notify the user or abort...

That's just moving a host implemantation issue into plugins. Allowing 
late events is almost like allowing late audio buffers. The only 
difference is that with events, it *looks* simple, whereas with 
audio, it's obvious that you'd need a different protocol. When you 
look closer at it, specifically at ramping, it becomes obvious that 
there isn't all that much of a difference.

I don't think plugins can do anything useful about late events that 
hosts can't fix by adjusting the timestamps, and as this never 
happens within a real time net in a single thread, it doesn't even 
have to be considered in most hosts. Only hosts that support soft RT 
connections between multiple threads or processes will need to deal 
with this, and they can do it inside the event gateways that are 
required anyway.


> but why do you want to rule out late events ?
> they can be handled in a sensible way.

I don't think they can be handled trivially, and I don't see any 
reason why the API or plugins should consider it at all. Whenever a 
host makes a connection that can result in late events, it has to 
make sure the events are adjusted as needed, so plugins get delayed 
or quantized events.


> i think this is similar to the soft mode of jack....

Maybe, but audio is not structured data, and thus, isn't as sensitive 
to glitches. You don't have to do anything special to "fix" drop-outs 
in audio streams without totally screwing up the data.


[...]
> i meant event feedbacks without a delay but with an event gate to
> abort in some condition...
>
> user can implement a for loop with this.
> if user is smart enough it can still be realtime safe...

We're talking about conditional "ping-pong" between plugins? Well, 
that's kind of tricky to implement with a block based callback model, 
since each plugin only gets to run once per block. A roundtrip always 
means one block of latency.


> > > (but if the code was smart enough it could solve
> > >  the cycle at the delay)
> >
> > Plugins won't and shouldn't care. Feedback is a host
> > implementation issue.
>
> hmm... ok... what about the for loop implemented with events?
> i dont like to give this idea up....

What exactly do you want to achieve with this? It sounds like this 
belongs on a much lower level than the plugin API... XAP plugin nets 
are not like "bytecode" with their own flow control.


> but this does not work with the XAP model...

Actually, I think it might, but I'm not sure what you're trying to do, 
or how it applies to audio and/or control processing.


[...]
> > There's simply no other way of doing it. It could theoretically
> > take *minutes* to load all the samples for a song. (At least if
> > they're on CDs in a CD-ROM changer...) What can the API or
> > plugins do about that...?
>
> in this case the delay would convert the delayed event
> to a delayed event and it would be immedietly processed
> when injected into the realtime thread again...

Of course, but that's not useful. No matter how it's handled; if the 
sampler isn't ready when I start playing notes, I'm screwed. The only 
useful feature beyond plugins being able to do non RT stuff in worker 
callbacks, would be the ability to tell whether a plugin is ready or 
not. The only safe solution is to wait until a plugin is ready before 
messing with it.


> > > the sample loading component would fire a worker thread,
> > > which inserts the event with the same timestamp
> > > as the incoming event... (at this point the timestamp is
> > > in the past)
> > > the evtdelay adjusts the timestamp to be in the future
> > > again.
> >
> > I don't see much point in this. How do you figure out the delay
> > value?
>
> user can control the value with a control in the panel...
> if user knows what he does he can set the delay to a
> working value... where no droputs ocur...

This is not possible, and IMNSHO it's completely pointless to even 
consider this on the API level. There is *no upper limit* to the time 
it may take to allocate some memory in a general purpose OS, such as 
Linux, Mac OS X or Windows. Deal with it, or switch to a complete 
RTOS with virtual memory disabled.


> delayed events could be reported to the engine making some led
> blink...

Sure, but who cares? Depending on soft RT features in a hard RT system 
makes the whole system essentially soft RT. I think a "READY" output 
for that LED would be sufficient and useful, but going further than 
that is just a futile atempt to make soft real time look harder than 
it is. If you want all hard RT, just never change soft RT controls 
during performance. Plugins that can't be used without doing that are 
either broken or not meant for real time operation.


> > > i take this approach for midi in also...
> > > (not implemented yet)
> > > the midi event has a timestamp from alsa, which corresponds
> > > to the past. without the delay it would be processed now..
> > > this would generate jitter.
> > > with the delay some latency is imposed but the jitter is away.
> > > the user can adjust the delay to his machine...
> >
> > That's very different. This is what you're expected to do with
> > incoming real time events, and the resulting constant latency is
> > strictly defined by the block size. That's all there is to it. It
> > has nothing to do with worker callbacks, delayed events or other
> > "tricks" to deal with plugins that have non-RT safe controls.
>
> i dont really see a difference... this is an event which comes
> from another thread. like the event which comes from the worker
> thread...

The worker thread belongs to the plugin, and is an internal 
implementational matter. Worker threads are meant specifically for 
jobs that are not RT safe. A plugin may not be able to process other 
events *at all* until the worker thread has finished. It might just 
sit there and track incomming events, so the rest of the net can keep 
running.

MIDI events OTOH (or other events that come from other threads), come 
from the outside world. Plugins don't care if they're late, and in 
fact, won't even know about it, since they can't see the original 
timestamps.

I think there's quite a difference between the two, especially since 
the first case is strictly related to soft real time, wherease the 
latter doesn't have to have anything to do with soft RT at all. The 
former is a plugin implementation thing, while the latter is not even 
visible to plugins.


[...]
> > > > > yes... But due to the event peeking code it would get the
> > > > > event 100ms before it is due. The event peeker is too
> > > > > complicated though.
> > > >
> > > > In fact, it's not even possible to implement in a generic
> > > > way. (See above.)
> > >
> > > i think i have found a method above...
> >
> > I don't think so. You can't see into the future, so you must use
> > a delay. When you're dealing with non-deterministic operations
> > (which is the only case where you need to mess with this stuff),
> > you can't figure out what a sufficient delay time would be. When
> > dealing with "live" input, delays are just not an option, as
> > they'd defeat the whole purpose of real time processing.
>
> well what with midi jitter ?
> midi event -> midiport -> timestamp -> kernel doing blabla.. ->
> user space midi thread -> real time thread...
>
> long path for the midi event... i suspect
> event bursts...

Sure, but it has nothing to do with peeking into the future. If you 
can't get MIDI events delivered in time, it's an OS and/or driver 
problem, and there isn't much we can do about it. (Short of fixing 
the OS and/or driver, of course.)

This is just a soft->hard RT gateway. Assume a delay that will be 
sufficient most of the time, and adjust late events if you get them. 
That's all there is to it. I don't see how this is related in any way 
to peeking in order to "prepare" nondeterministic plugins.


> > [...]
> >
> > > Yes... but galan has no stop. It is always running.
> >
> > Just like hardware synths - and XAP should work the same way. The
> > way I see it, very few hosts have valid reasons to ever stop
> > processing.
> >
> > What I'm talking about is the *sequencer*, which will always have
> > transport control (with stop, start, rewind etc), and that's
> > where this comes in. When you load a song, you'll have to
> > initialize the net before you can start playing, just as with
> > external hardware samplers that need to grind and rattle some
> > before you can play.
>
> yes but constuctors are not executed in the realtime thread.
> the plugin only gets inserted into the net
> after its constructor was run..

I'm not talking about constructors, but about loading presets and 
changing controls. That's nondeterministic stuff, so if you want to 
be sure you won't screw up half your intro, you simply have to wait 
for it to finish before you start playing.


> > > But a change sample will fire a worker and leave the old sample
> > > until the new sample arrives.
> >
> > Yes, but that's just a plugin implementation detail. The only API
> > implication it has is that such controls should be marked as "may
> > not respond instantly" - and not even that is strictly required.
> > (MIDI samplers don't have such a feature, AFAIK.)
>
> ah... the execution path of an event can enter the realtime thread,
> and leave it again into another thread...

Not quite, since the event itself never gets to the worker thread. The 
event is received by the plugin, which sends the *job* off to a 
worker thread. When the worker thread is done, the host sends a 
"result" event to the plugin, which can then go on with any further, 
RT safe operations.


> if the realtime thread advanced to much then
> the event would have a timestamp in the past.

No, becaues there would be no events at all outsid the RT thread. The 
"result" event is generated by the host as soon as it finds out that 
the worker thread has finished.


> events with timestamps < gen_get_sample_time() are always processed
> first:  better late than never.

Well, you could say the result events are handled like this, but it's 
done by the host (by picking a suitable timestamp), and the events 
aren't really late. They just tell the plugin that it's worker thread 
has finished.


[...]
> > > do want to have audiality running in the kernel ?
> >
> > Actually, the "second generation" of Audiality (the "real" one is
> > the third) was intended to run in kernel space, but that was
> > before Linux/lowlatency. Back then, the only way to get solid RT
> > performance on Linux was through RTLinux, so that's what I went
> > for.
>
> did you ever install RTLinux ?
> is that stress ?

It was quite some time ago I used it, but I had no major problems. 
It's just a kernel patch and some modules. These days, I believe 
there are some shared libs as well.

I installed RTAI recently, and the only issue with that was to find a 
compiler that was supported by both RTAI and my kernel. I ended up 
compiling my own compiler, and then everything worked fine. (This was 
on an embedded SBC system. Yes, it did take ages to build the 
compiler... :-)


> i am experimenting with 2.5.x kernel with alsa inside
> just had galan running without glitches...

Well, unfortunately, RTLinux or RTAI won't help, at least not without 
some hacking. You could probably hack galan to run in user space via 
LXRT ("RTAI in user space"), but you'd have to do some tricks with 
the I/O, as you can't use Linux drivers directly without the RT going 
soft. This is the major reason why I dropped RTLinux for audio when 
Linux/lowlatency showed up.


> > Anyway, both RTLinux and RTAI can schedule hard RT threads in
> > user space these days, so there's still no need to run in kernel
> > space, even if you need lower latency than Linux/lowlatency can
> > provide.
>
> very nice i should really try one of them...

Sure, but be prepared to either port drivers, or use mmap() mode and a 
PLL locked to the sound card IRQ. The latter shouldn't be too 
complicated, but I haven't bothered with it for various reasons.


[...building Audiality 0.1.0...]

You have ALSA 0.9.x, but this version of Audiality only supports 
0.5.x. It's just for raw MIDI anyway, so you might as well use OSS 
emulation. './configure --without-alsa'.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
   --- http://olofson.net --- http://www.reologica.se ---