[linux-audio-dev] XAP spec - early scribbles

[torbenh at gmx.de]

On Thu, Feb 27, 2003 at 08:18:02PM +0100, David Olofson wrote:
> On Thursday 27 February 2003 07.47, torbenh at gmx.de wrote:
> [...]
> > > There's only one call; process(). (Or run(), as some prefer to
> > > call it.) This is called once for each block, whether or not the
> > > plugin processes audio. Might sound odd, but considering that the
> > > unit for event timestamps is audio frames, it turns out to be
> > > really rather logical - in theory as will as implementation.
> >
> > Yeah youre right.
> > But then i have to implement graph ordering.
> > which galan does not do at the Moment.
> > I will also have to isolate the OpenGL stuff from
> > the audio processing... Well i have to do that
> > anyway...
> 
> Yeah. Without strict graph ordering, you get effectively "random" 
> latency in every connection. At best, it depends in non-obvious ways 
> on the order in which plugins are connected, or something like that. 
> I don't think that's a good idea - not even for blockless processing.

random latency ? how do you mean that ?

see current implementation...

here is the output generator of the galan gain:


PRIVATE gboolean output_generator(Generator *g, SAMPLE *buf, int buflen) {
  Data *data = g->data;
  int i;

  if (!gen_read_realtime_input(g, SIG_INPUT, -1, buf, buflen))
    return FALSE;

  /* I'm doing this check after reading the input just to keep the
     timing in sync. Just a minor niggle. */
  if (data->gain == 0.0)
    return FALSE;

  if (data->gain != 1.0) {
    for (i = 0; i < buflen; i++)
      buf[i] *= data->gain;
  }

  return TRUE;
}

oh i did not notice it was calling the read_realtime input...
but this can be eliminated if all critical components would
check the sample time so they would do the right thing even
if they had not been called for some blocks...

so this call only compensates for buggy other generators...

here comes gen_read_realtime_input:

/**
 * \brief read an input of Type SIG_FLAG_REALTIME into the \buffer.
 *
 * This function is used to get the realtime signal data.
 * 
 * \param g this Generator.
 * \param index which input connector
 * \param attachment_number which of the connected Generator s (should normally be -1 for all)
 * \param buffer where to put the data.
 * \param buflen how many samples should be read.
 *
 * \return TRUE if something was put into the \a buffer.
 */

PUBLIC gboolean gen_read_realtime_input(Generator *g, gint index, int attachment_number,
					SAMPLE *buffer, int buflen) {
  /* Sanity checks */
  g_return_val_if_fail(index < g->klass->in_sig_count && index >= 0, FALSE);
  g_return_val_if_fail((g->klass->in_sigs[index].flags & SIG_FLAG_REALTIME) != 0, FALSE);

  if (attachment_number == -1 && g_list_next(g->in_signals[index]) == NULL)
    attachment_number = 0;	/* one input? don't bother summing. */

  if (attachment_number == -1) {
    SAMPLE tmp[MAXIMUM_REALTIME_STEP];
    GList *lst = g->in_signals[index];
    gboolean result = FALSE;

    memset(buffer, 0, sizeof(SAMPLE) * buflen);
    while (lst != NULL) {
      EventLink *el = lst->data;
      lst = g_list_next(lst);

      if (gen_read_realtime_output(el->src, el->src_q, tmp, buflen)) {
	int i;

	for (i = 0; i < buflen; i++)
	  buffer[i] += tmp[i];
	result = TRUE;
      }
    }

    return result;
  } else {
    GList *input_list = g_list_nth(g->in_signals[index], attachment_number);
    EventLink *el;

    if (input_list == NULL) {

      // oh what is this ?
      // go away :) if plugin gets false it can not assume buffer is valid.
      
      memset(buffer, 0, buflen * sizeof(SAMPLE));
      return FALSE;
    }

    el = (EventLink *) input_list->data;
    return gen_read_realtime_output(el->src, el->src_q, buffer, buflen);
  }
}

/**
 * \brief read realtime output from a generator and cache it if its necessarry.
 *
 * I dont think someone would use this function. It is used by gen_read_realtime_input()
 * to obtain the data from the individual Generator s and cache it if it will be read multiple
 * times.
 *
 * \param g the Generator to be read out.
 * \param index connector number.
 * \param buffer where the data should be placed.
 * \param buflen how many SAMPLE s should be read.
 *
 * \return TRUE if data could be read.
 * 
 */

PUBLIC gboolean gen_read_realtime_output(Generator *g, gint index, SAMPLE *buffer, int buflen) {
  /* Sanity checks */
  g_return_val_if_fail(index < g->klass->out_sig_count && index >= 0, FALSE);
  g_return_val_if_fail((g->klass->out_sigs[index].flags & SIG_FLAG_REALTIME) != 0, FALSE);

  if (g_list_next(g->out_signals[index]) == NULL)
    /* Don't bother caching the only output. */
    return g->klass->out_sigs[index].d.realtime(g, buffer, buflen);
  else {
    /* Cache for multiple outputs... you never know who'll be reading you, or how often */
    if (g->last_buffers[index] == NULL || g->last_sampletime < gen_get_sampletime()) {
      /* Cache is not present, or expired */
      if (g->last_buffers[index] != NULL)
	free(g->last_buffers[index]);
      g->last_buffers[index] = malloc(sizeof(SAMPLE) * buflen);
      g->last_buflens[index] = buflen;
      g->last_sampletime = gen_get_sampletime();
      g->last_results[index] =
	g->klass->out_sigs[index].d.realtime(g, g->last_buffers[index], buflen);
    } else if (g->last_buflens[index] < buflen) {
      /* A small chunk was read. Fill it out. */
      SAMPLE *newbuf = malloc(sizeof(SAMPLE) * buflen);
      int oldlen = g->last_buflens[index];

      if (g->last_results[index])
	memcpy(newbuf, g->last_buffers[index], sizeof(SAMPLE) * g->last_buflens[index]);
      else
	memset(newbuf, 0, sizeof(SAMPLE) * g->last_buflens[index]);
      free(g->last_buffers[index]);
      g->last_buffers[index] = newbuf;
      g->last_buflens[index] = buflen;
      g->last_results[index] = 
	g->klass->out_sigs[index].d.realtime(g, &g->last_buffers[index][oldlen], buflen - oldlen);
    }

    if (g->last_results[index])
      memcpy(buffer, g->last_buffers[index], buflen * sizeof(SAMPLE));
    return g->last_results[index];
  }
}

ok... most of the code has todo with caching for multiple connections...
in the 1:1 case both functions only call the right function.


one advantage is with silent sub nets....


in the current galan implementation a call
to gen_read_realtime_input( blabla )
will immediately return FALSE if the input is connected
to a gain set to zero (well the gain returns false and should not
traverse its inputs further..)


also in some complex event processing nets
in XAP every plugin gets process call normally only realizing:
  no input events -> nothing todo

in galan a plugin can register itself to get realtime callbacks...
(like XAP process callback)

But it does not need to... i will have to do some profiling
on galan if i have the time...


these calls are not ordered ATM but this could be fixed of course.

when migrating away from the global eventqueue to localized ones
it should be possible to do local buffer splitting in the gen_read_realtime_input()




> 
> 
> [...]
> > > This suggests to me that "look-ahead" is a special case interface
> > > for some specific cases; not an idea to build a real time API
> > > upon.
> >
> > yes...
> > a sequencer could have a clock and a prepare-clock
> > and then outs and prepare-outs....
> > this fits on top of XAP.
> 
> Sure, but I can't quite see where it would be useful. It's just a 
> means of squeezing inherently non-real time features into a strict 
> real time system. Real time controls should never need this, because 
> that would make it impossible no use the plugins in *true* real time 
> systems.

ok... but i see realtime as a subset of offline audio processing...
there are some things you must not do in a realtime system.
but you can do these in a nonRT system...

you can build realtime systems with C although you can also do
while(1) ;

i see galan as a programming language.
so i dont want to lock the user in a system where he can only
build realtime systems. galan should warn the user if he built
something not realtime safe though... but it should not be an error....

> 
> 
> > hmmm... but what if
> >
> > an event gate:
> >
> > if it gets an event on input
> >   if the last event on tocomp == the last event on comp
> >     output event from input.
> [...]
> > this is not very excact but it is possible to build such
> > a mesh in galan. hope you get what i mean from the text....
> 
> I'm not sure... Is this basically about splitting actions up into two 
> events; one "this is what we'll do" event, and one "do it now" event?

yes sort of...
the idea is to put plugins doing the prepare stuff in front of
the delay so they could safely fire a worker thread...
when the worker thread finishes... the delay adjusts
the timestamp compensating for the non determinism of
the worker thread...

i see this as a method of trying to guarantee some time
the worker thread has...

but if the event is late after delaying it, the delay is not
calibrated correctly...
but it is not so bad because realtime processing was not interrupted...


> 
> 
> [...]
> > in XAP this event feedback would already be illegal.
> 
> No, feedback isn't illegal in any way. It's just "late events" that 
> are illegal; only events for this or future blocks must even be found 
> in an input event queue.

why ?
the semantic of a "late event" is process me as fast as you can
i am late. plugins could report that they received a late event
to the host so it could notify the user or abort...
but why do you want to rule out late events ?
they can be handled in a sensible way.

i think this is similar to the soft mode of jack....

> 
> If you set up a feedback loop, you have to make sure events are 
> delayed appropriately, so timestamps remain valid. This automatically 
> eliminates block size quantization issues, as the delay must be at 
> least the duration of one block. If you need shorter feedback 
> latency, use smaller blocks. If you need less than one sample, well, 
> then it's time you think about implementing a new plugin. You'll 
> probably have to use approximations, prediction filters and/or 
> oversampling.

i meant event feedbacks without a delay but with an event gate to
abort in some condition...

user can implement a for loop with this.
if user is smart enough it can still be realtime safe...

> 
> 
> > (but if the code was smart enough it could solve
> >  the cycle at the delay)
> 
> Plugins won't and shouldn't care. Feedback is a host implementation 
> issue.

hmm... ok... what about the for loop implemented with events?
i dont like to give this idea up....

but this does not work with the XAP model...
i see your point though...

> 
> 
> > in galan this is legal but provides the user with
> > the possibility to build meshes which are not realtime
> > safe... you can also build for loops etc...
> 
> Right. This is a fact of nature we have to deal with. Feedback loops 
> *always* have some latency (be it nanoseconds or milliseconds), and 
> there's nothing we can do about it, except make sure we know where 
> those latencies are, and consider them when tuning our designs.
> 
> 
> > if there was a component which converted a string event
> > to a sample event, this could be connected before the
> > delay.
> >
> > so the sample would already be loaded when it is time
> > for the delay to let the event pass...
> 
> There's simply no other way of doing it. It could theoretically take 
> *minutes* to load all the samples for a song. (At least if they're on 
> CDs in a CD-ROM changer...) What can the API or plugins do about 
> that...?

in this case the delay would convert the delayed event
to a delayed event and it would be immedietly processed
when injected into the realtime thread again...
> 
> 
> > the sample loading component would fire a worker thread,
> > which inserts the event with the same timestamp
> > as the incoming event... (at this point the timestamp is
> > in the past)
> > the evtdelay adjusts the timestamp to be in the future
> > again.
> 
> I don't see much point in this. How do you figure out the delay value?

user can control the value with a control in the panel...
if user knows what he does he can set the delay to a
working value... where no droputs ocur...

delayed events could be reported to the engine making some led blink...



> 
> 
> > i take this approach for midi in also...
> > (not implemented yet)
> > the midi event has a timestamp from alsa, which corresponds
> > to the past. without the delay it would be processed now..
> > this would generate jitter.
> > with the delay some latency is imposed but the jitter is away.
> > the user can adjust the delay to his machine...
> 
> That's very different. This is what you're expected to do with 
> incoming real time events, and the resulting constant latency is 
> strictly defined by the block size. That's all there is to it. It has 
> nothing to do with worker callbacks, delayed events or other "tricks" 
> to deal with plugins that have non-RT safe controls.

i dont really see a difference... this is an event which comes
from another thread. like the event which comes from the worker thread...

> 
> 
> > > > this seems to be not feasible, because the event peeker would
> > > > be fairly complex it must not follow the code path of the clock
> > > > posting events to itself because of the infinite loop etc...
> > > > not taking into account bigger event feedback loops....
> > >
> > > The "event peeker" has to be part of the clock and send events
> > > that are part of a special protocol for this task.
> > >
> > > We have something similar in XAP, meant for hard disk recorders
> > > and other plugins that need pre-buffering time before they can
> > > start playing from a random song position. A sequencer (or
> > > rather, timeline) can inform such plugins about loop and jump
> > > points beforehand, so that specific pre-buffering can be used to
> > > make it possible to jump to these points with zero latency.
> >
> > ok.. it is time for me to look at the XAP specs...
> > where are they ?
> > as it seems you are already defining the higher levels...
> 
> We have discussed a few of the higher level concepts, but there's 
> still a lot of work to do in that area.
> 
> As to the actual spec document, I have yet to see it myself! :-) So 
> far, all that's available is the list archives (ouch!) and the 
> terminology document on the site:
> 
> 	http://xap-plugins.org

i will have a look...
> 
> 
> Tim Hockin is working on the first preliminary spec. For now, you I 
> suppose you can bug us with questions. If you try hard enough, we 
> might give in and release what we have. ;-)

:-) ok... 
at the moment the only advantage of XAP i see is that
the compressor example works...

but this could work in galan also with localized eventqueues...
i believe..

> 
> 
> [...]
> > > > yes... But due to the event peeking code it would get the event
> > > > 100ms before it is due. The event peeker is too complicated
> > > > though.
> > >
> > > In fact, it's not even possible to implement in a generic way.
> > > (See above.)
> >
> > i think i have found a method above...
> 
> I don't think so. You can't see into the future, so you must use a 
> delay. When you're dealing with non-deterministic operations (which 
> is the only case where you need to mess with this stuff), you can't 
> figure out what a sufficient delay time would be. When dealing with 
> "live" input, delays are just not an option, as they'd defeat the 
> whole purpose of real time processing.

well what with midi jitter ?
midi event -> midiport -> timestamp -> kernel doing blabla.. -> user space midi thread -> real time thread...

long path for the midi event... i suspect
event bursts... 

> 
> [...]
> > Yes... but galan has no stop. It is always running.
> 
> Just like hardware synths - and XAP should work the same way. The way 
> I see it, very few hosts have valid reasons to ever stop processing.
> 
> What I'm talking about is the *sequencer*, which will always have 
> transport control (with stop, start, rewind etc), and that's where 
> this comes in. When you load a song, you'll have to initialize the 
> net before you can start playing, just as with external hardware 
> samplers that need to grind and rattle some before you can play.

yes but constuctors are not executed in the realtime thread.
the plugin only gets inserted into the net 
after its constructor was run..

> 
> 
> > But a change sample will fire a worker and leave the old sample
> > until the new sample arrives.
> 
> Yes, but that's just a plugin implementation detail. The only API 
> implication it has is that such controls should be marked as "may not 
> respond instantly" - and not even that is strictly required. (MIDI 
> samplers don't have such a feature, AFAIK.)

ah... the execution path of an event can enter the realtime thread,
and leave it again into another thread...

if the realtime thread advanced to much then 
the event would have a timestamp in the past.
events with timestamps < gen_get_sample_time() are always processed first:  better late than never.




> 
> [...]
> > do want to have audiality running in the kernel ?
> 
> Actually, the "second generation" of Audiality (the "real" one is the 
> third) was intended to run in kernel space, but that was before 
> Linux/lowlatency. Back then, the only way to get solid RT performance 
> on Linux was through RTLinux, so that's what I went for.

did you ever install RTLinux ?
is that stress ?

i am experimenting with 2.5.x kernel with alsa inside
just had galan running without glitches...
> 
> Anyway, both RTLinux and RTAI can schedule hard RT threads in user 
> space these days, so there's still no need to run in kernel space, 
> even if you need lower latency than Linux/lowlatency can provide.

very nice i should really try one of them...

> 
> 
> [...]
> > too bad i missed the XAP discussions...
> > but i had to waste my time with 1.Semester Computer Science
> > (at least i heard Math 3 which was kind of interresting
> >
> > it gave me a different look (more algebraic than the engeneers)
> > on the Analysis
> 
> Can't see how that can be wasted time. :-)
> 
> 
> [...ebuild...]
> 
> Sorry! I should have know that, being that I've considered using 
> Gentoo. The name just didn't ring a bell somehow, and the first 
> document I found didn't say what the tool is *for*... :-)

gentoo is very nice...
it can build ardour from cvs with
emerge ardour 

and you always get all header files from libraries just as if you
configure; make; make installed them....

and if you do "emerge update world" quite frequently your CPU will
know why it is on the world :)



> Anyway, the explanation is simple; I don't support packages at all.
> Whatever is there is old contributed stuff that came from Kobo Deluxe, 
> which is where Audiality was born. It most probably doesn't work. I 
> should clean up my scripts...

well it did not compile on first try...

In file included from a_midi.c:33:
/usr/include/sys/asoundlib.h:1:2: warning: #warning This header is deprecated, use <alsa/asoundlib.h> instead.
a_midi.c: In function `midi_open':
a_midi.c:190: `SND_RAWMIDI_OPEN_INPUT' undeclared (first use in this function)
a_midi.c:190: (Each undeclared identifier is reported only once
a_midi.c:190: for each function it appears in.)
a_midi.c:191: `SND_RAWMIDI_OPEN_NONBLOCK' undeclared (first use in this function)

i will investigate later...

> //David Olofson - Programmer, Composer, Open Source Advocate
> 
> .- The Return of Audiality! --------------------------------.
> | Free/Open Source Audio Engine for use in Games or Studio. |
> | RT and off-line synth. Scripting. Sample accurate timing. |
> `-----------------------------------> http://audiality.org -'
>    --- http://olofson.net --- http://www.reologica.se ---
> 

-- 
torben Hohn
http://galan.sourceforge.net -- The graphical Audio language