4 Mar
2003
4 Mar
'03
9:06 a.m.
On Fri, Feb 28, 2003 at 07:05:19PM +0100, David Olofson wrote:
yeah i see your point...
how about a flag indicating "dont call my process() from the realtime
thread, but from another thread"
the event sending in this case needs to be different but this could
be a different function/macro which needs to be called in this case.
note that in galan i have the opengl rendering components which receive
events also. I would be nice if this could be handled by XAP...
yes this is correct.
but if the plugin could decide by itself what to do with a late
event everything would be fine.
yes... and how is this decision made ?
in my proposal the user would wire a delay or a quantizer into the net
and everything would be fine.
in your proposal you would have some logic in the host decide what to do
generating complexity in the host as this question must reach the user
somehow....
yes...
but if the system was robust enough to handle an event dropout it would
be ok. also the user could guard himself against the dropout case...
with an eventgate of some sort.
hmm... ok... what about the for loop implemented with events?
i dont like to give this idea up....
What exactly do you want to achieve with this? It sounds like this
belongs on a much lower level than the plugin API... XAP plugin nets
are not like "bytecode" with their own flow control.
ok... i just like the idea of galan beeing turing complete :)
well i dont really know what i want to achieve with this.
i have not yet spent a day experimenting with complex
event logic...
i spend too much time on thinking about our current discussions...
user can control the value with a control in the panel...
if user knows what he does he can set the delay to a
working value... where no droputs ocur...
This is not possible, and IMNSHO it's completely pointless to even
consider this on the API level. There is *no upper limit* to the time
it may take to allocate some memory in a general purpose OS, such as
Linux, Mac OS X or Windows. Deal with it, or switch to a complete
RTOS with virtual memory disabled.
if the user knows there are still 512 MB free there is an upper bound.
ah.. there are soft RT and real RT controls ?
is that a hint on the control ?
i dont really see a difference... this is an event which comes
from another thread. like the event which comes from the worker
thread...
The worker thread belongs to the plugin, and is an internal
implementational matter. Worker threads are meant specifically for
jobs that are not RT safe. A plugin may not be able to process other
events *at all* until the worker thread has finished. It might just
sit there and track incomming events, so the rest of the net can keep
running.
MIDI events OTOH (or other events that come from other threads), come
from the outside world. Plugins don't care if they're late, and in
fact, won't even know about it, since they can't see the original
timestamps.
I think there's quite a difference between the two, especially since
the first case is strictly related to soft real time, wherease the
latter doesn't have to have anything to do with soft RT at all. The
former is a plugin implementation thing, while the latter is not even
visible to plugins.
hmm... i see your point somehow...
but i am still not convinced...
I'm not talking about constructors, but about loading presets and
changing controls. That's nondeterministic stuff, so if you want to
be sure you won't screw up half your intro, you simply have to wait
for it to finish before you start playing.
yes ... correct...
but how about streaming components ?
the ogg_ra component has an input event triggering the loading
of the next buffer. in galan 0.3.x there is a worker thread
exchanging 2 buffers with the realtime thread to make this
operation realtime safe. but this does not handle skipping
elegantly....
i would rather implement the loading in a component getting
a time event and then emiting an array of audio data...
this way i can prebuffer in the eventdelay...
if the event does not come through in time the
old audio data will still sit in the sample component
and this will be played then...
just like an xrun where the soundcard keeps playing...
by adjusting the evtdelay the user could enable this ogg player
to handle most skips without an xrun ( note that this is only
an xrun within the mesh-maschine...) it can be detected by the mesh,
and it could take precautions to not make it sound so bad...
(eg turn on a low pass so that the pop wont be loud)
ah... the execution path of an event can enter the realtime thread,
and leave it again into another thread...
Not quite, since the event itself never gets to the worker thread. The
event is received by the plugin, which sends the *job* off to a
worker thread. When the worker thread is done, the host sends a
"result" event to the plugin, which can then go on with any further,
RT safe operations.
but then the delay must be part of the plugin...
and i think that this is not necessary...
i think the mesh would be easier to understand if
the delay was an explicit element of the net and not hidden
in some component.
hmmm... what about the opengl thread ?
i need a way to have different threads and dont want to process all
events in the realtime thread which is obvious, isnt it.
This also a requirement for the graph sorter...
how do we model this ?
now the host destroys a timestamp which could be fixed by a delay.
i dont want delays to be special components... the simply arent.
ok... so i think i wont bother...
Sure, but be prepared to either port drivers, or use mmap() mode and a
PLL locked to the sound card IRQ. The latter shouldn't be too
complicated, but I haven't bothered with it for various reasons.
i have an event PLL in galan :)
it could be possible to model the output as a randomaccess input
filling it when the clock event gets through...
hmm... i should think about this a little more...
this would make up some nice lowlevel component set :)
On Friday 28 February 2003 09.20, torbenh(a)gmx.de
wrote:
yes and because a nonRT system can do more things it is a superset
of RT. (but this is just being picky :)
ok... but i see realtime as a subset of offline
audio processing...
It is not. Real time applications have some very strict requirements
that no other applications have. Sinci there is no way around this
fact, we just have to deal with it, and design the rest of the system
around the requirements.
there are some things you must not do in a
realtime system.
but you can do these in a nonRT system...
Exactly - and that is why you must design in a way that allows RT
hosts and plugins to operate without non RT safe actions. you can build realtime systems with C although
you can also do
while(1) ;
i see galan as a programming language.
so i dont want to lock the user in a system where he can only
build realtime systems. galan should warn the user if he built
something not realtime safe though... but it should not be an
error....
Well, it's not really an error to implement a XAP plugin that won't
run in real time and/or is nondeterministic. However, I don't quite
see what you could do in an RT capable *host* that inherently cannot
run in RT. Multi-pass subnets with conditional loops...? Can you give
some examples?
I realize that non-linear time processing falls into this category,
but I don't see how that is relevant to a real time plugin API. I can
see how it would be *possible* to support it in a plugin API that
also does real time processing, but I think designing something like
that is asking for serious trouble. There are just too many conflicts
between these two ways of working.
[...]
Right - but then, why mess with this on the API level, and why use a
delay at all? Well, I can see the jitter reduction advantage, but I
don't think it's worth it. Just design plugins so that you can tell
them what you're going to do first, and then have guaranteed RT
response.
how is this guaranteed ?
when i tell a plugin it should load sample X
it might take hours to download this url via modem.....
it seems like you expect the user to know that and dont
use the sample until its downloaded.
the evtdelay and an evtgate are some of the plugins who
could handle delayed events and i dont see the reasons
to rule these out.
the plugin must do its own buffer splitting, process ramps etc...
why dont you want it to do
// This if() needs to be a nice macro for fixing wrapping
// or #define SAMPLETIME gint64
if( event->timestamp < current_time )
drop_event_if_i_dont_like_it()
even late ramp events could be fixed... if the host did
not destroy its timestamp by setting it to current_time...
what are your reasons for this ?
does it have to do with timestamp wrapping ?
this could be fixed trivially...
I'm
not sure... Is this basically about splitting actions up into
two events; one "this is what we'll do" event, and one "do it
now" event?
yes sort of...
the idea is to put plugins doing the prepare stuff in front of
the delay so they could safely fire a worker thread...
when the worker thread finishes... the delay adjusts
the timestamp compensating for the non determinism of
the worker thread...
i see this as a method of trying to guarantee some time
the worker thread has...
but if the event is late after delaying it, the delay is not
calibrated correctly...
but it is not so bad because realtime processing was not
interrupted...
For example, the delay could have a "max delay time" control for
buffer allocation, and a "delay time" control for setting the actual
delay. That way, you can use the same plugin for ms delays as well as
delays of several seconds, without hard-coding the plugin to some
arbitrary huge buffer size.
max delay time is not realtime safe...
how shall such an event be handled ?
[...]
why ?
the semantic of a "late event" is process me as fast as you can
i am late.
Sure, but timestamps wrap, and as a results, they wrap into the future
if they arrive late. This can be "fixed", but it requires that we
agree on a specific maximum allowed time an event may be late.
no problem MAX_TIMESTAMP - current_blocksize
future events are not delivered to the plugins.
in XAP this event feedback would already be illegal.
No, feedback isn't illegal in any way. It's just "late events"
that are illegal; only events for this or future blocks must even
be found in an input event queue.
More importantly though; it's not possible to handle control ramping
properly, unless timestamps are respected as part of the data.
"Process ASAP" results in quantization, and that will screw up chains
of ramps. (XAP ramps are just "aim point" commands, and nothing is
guaranteed after an aim point is passed. Plugins are not required to
stop ramping automatically, but can expect to receive a new event in
time.)
yes... normal plugins can drop this late events or try to interpolate
to the current time.
there are not much plugins capable of handling late events
but a delay could handle them.
plugins could report that they received a late
event
to the host so it could notify the user or abort...
That's just moving a host implemantation issue into plugins. Allowing
late events is almost like allowing late audio buffers. The only
difference is that with events, it *looks* simple, whereas with
audio, it's obvious that you'd need a different protocol. When you
look closer at it, specifically at ramping, it becomes obvious that
there isn't all that much of a difference.
I don't think plugins can do anything useful about late events that
hosts can't fix by adjusting the timestamps, and as this never
happens within a real time net in a single thread, it doesn't even
have to be considered in most hosts. Only hosts that support soft RT
connections between multiple threads or processes will need to deal
with this, and they can do it inside the event gateways that are
required anyway. but why do you want to rule out late events ?
they can be handled in a sensible way.
I don't think they can be handled trivially, and I don't see any
reason why the API or plugins should consider it at all. Whenever a
host makes a connection that can result in late events, it has to
make sure the events are adjusted as needed, so plugins get delayed
or quantized events. i think this is similar to the soft mode of
jack....
Maybe, but audio is not structured data, and thus, isn't as sensitive
to glitches. You don't have to do anything special to "fix" drop-outs
in audio streams without totally screwing up the data.
[...]
yes... i see the advantages of running once per block...
the could be also done by the host compiling a graph into
an XAP plugin ... i am fine with this method as i see a way
out of the misery for me :)
i meant event feedbacks without a delay but with
an event gate to
abort in some condition...
user can implement a for loop with this.
if user is smart enough it can still be realtime safe...
We're talking about conditional "ping-pong" between plugins? Well,
that's kind of tricky to implement with a block based callback model,
since each plugin only gets to run once per block. A roundtrip always
means one block of latency. (but if the code was smart enough it could solve
the cycle at the delay)
Plugins won't and shouldn't care. Feedback is a host
implementation issue. [...]
Of course, but that's not useful. No matter how it's handled; if the
sampler isn't ready when I start playing notes, I'm screwed. The only
useful feature beyond plugins being able to do non RT stuff in worker
callbacks, would be the ability to tell whether a plugin is ready or
not. The only safe solution is to wait until a plugin is ready before
messing with it.
how do you know its ready ?
There's simply no other way of doing it. It could theoretically
take *minutes* to load all the samples for a song. (At least if
they're on CDs in a CD-ROM changer...) What can the API or
plugins do about that...?
in this case the delay would convert the delayed event
to a delayed event and it would be immedietly processed
when injected into the realtime thread again... the sample loading component would fire a worker
thread,
which inserts the event with the same timestamp
as the incoming event... (at this point the timestamp is
in the past)
the evtdelay adjusts the timestamp to be in the future
again.
I don't see much point in this. How do you figure out the delay
value? delayed events could be reported to the engine
making some led
blink...
Sure, but who cares? Depending on soft RT features in a hard RT system
makes the whole system essentially soft RT. I think a "READY" output
for that LED would be sufficient and useful, but going further than
that is just a futile atempt to make soft real time look harder than
it is. If you want all hard RT, just never change soft RT controls
during performance. Plugins that can't be used without doing that are
either broken or not meant for real time operation. i take this approach for midi in also...
(not implemented yet)
the midi event has a timestamp from alsa, which corresponds
to the past. without the delay it would be processed now..
this would generate jitter.
with the delay some latency is imposed but the jitter is away.
the user can adjust the delay to his machine...
That's very different. This is what you're expected to do with
incoming real time events, and the resulting constant latency is
strictly defined by the block size. That's all there is to it. It
has nothing to do with worker callbacks, delayed events or other
"tricks" to deal with plugins that have non-RT safe controls.
[...]
well what with midi jitter ?
midi event -> midiport -> timestamp -> kernel doing blabla.. ->
user space midi thread -> real time thread...
long path for the midi event... i suspect
event bursts...
Sure, but it has nothing to do with peeking into the future. If you
can't get MIDI events delivered in time, it's an OS and/or driver
problem, and there isn't much we can do about it. (Short of fixing
the OS and/or driver, of course.)
This is just a soft->hard RT gateway. Assume a delay that will be
sufficient most of the time, and adjust late events if you get them.
That's all there is to it. I don't see how this is related in any way
to peeking in order to "prepare" nondeterministic plugins.
> > yes... But due to the event peeking code it
would get the
> > event 100ms before it is due. The event peeker is too
> > complicated though.
>
> In fact, it's not even possible to implement in a generic
> way. (See above.)
i think i have found a method above...
I don't think so. You can't see into the future, so you must use
a delay. When you're dealing with non-deterministic operations
(which is the only case where you need to mess with this stuff),
you can't figure out what a sufficient delay time would be. When
dealing with "live" input, delays are just not an option, as
they'd defeat the whole purpose of real time processing. [...]
yes but constuctors are not executed in the realtime thread.
the plugin only gets inserted into the net
after its constructor was run.. Yes... but galan has no stop. It is always
running.
Just like hardware synths - and XAP should work the same way. The
way I see it, very few hosts have valid reasons to ever stop
processing.
What I'm talking about is the *sequencer*, which will always have
transport control (with stop, start, rewind etc), and that's
where this comes in. When you load a song, you'll have to
initialize the net before you can start playing, just as with
external hardware samplers that need to grind and rattle some
before you can play. But a change sample will fire a worker and leave the
old sample
until the new sample arrives.
Yes, but that's just a plugin implementation detail. The only API
implication it has is that such controls should be marked as "may
not respond instantly" - and not even that is strictly required.
(MIDI samplers don't have such a feature, AFAIK.) if the realtime thread advanced to much then
the event would have a timestamp in the past.
No, becaues there would be no events at all outsid the RT thread. The
"result" event is generated by the host as soon as it finds out that
the worker thread has finished. events with timestamps < gen_get_sample_time()
are always processed
first: better late than never.
Well, you could say the result events are handled like this, but it's
done by the host (by picking a suitable timestamp), and the events
aren't really late. They just tell the plugin that it's worker thread
has finished.
[...]
did you ever install RTLinux ?
is that stress ?
It was quite some time ago I used it, but I had no major problems.
It's just a kernel patch and some modules. These days, I believe
there are some shared libs as well.
I installed RTAI recently, and the only issue with that was to find a
compiler that was supported by both RTAI and my kernel. I ended up
compiling my own compiler, and then everything worked fine. (This was
on an embedded SBC system. Yes, it did take ages to build the
compiler... :-)
hehe :)
do want to have audiality running in the kernel ?
Actually, the "second generation" of Audiality (the "real" one is
the third) was intended to run in kernel space, but that was
before Linux/lowlatency. Back then, the only way to get solid RT
performance on Linux was through RTLinux, so that's what I went
for. i am experimenting with 2.5.x kernel with alsa
inside
just had galan running without glitches...
Well, unfortunately, RTLinux or RTAI won't help, at least not without
some hacking. You could probably hack galan to run in user space via
LXRT ("RTAI in user space"), but you'd have to do some tricks with
the I/O, as you can't use Linux drivers directly without the RT going
soft. This is the major reason why I dropped RTLinux for audio when
Linux/lowlatency showed up. Anyway,
both RTLinux and RTAI can schedule hard RT threads in
user space these days, so there's still no need to run in kernel
space, even if you need lower latency than Linux/lowlatency can
provide.
very nice i should really try one of them...
[...building Audiality 0.1.0...]
You have ALSA 0.9.x, but this version of Audiality only supports
0.5.x. It's just for raw MIDI anyway, so you might as well use OSS
emulation. './configure --without-alsa'.
ok... it compiled... what do i do now ?
need to look more into it...
any exmaples for it ?
--
torben Hohn
http://galan.sourceforge.net -- The graphical Audio language