[linux-audio-dev] XAP spec - early scribbles

[David Olofson]

On Wednesday 05 February 2003 23.41, Tim Hockin wrote:
> > Well, you know my opinion about what controls plugins.... ;-)
>
> how about:
>
>   * Host
[...]
>   * Control:
[...]
>   * EventQueue

Good.


[...]
> > Yep. The effect of this might be cool enough to deserve a
> > comment;
>
> it is commented in the details section :)

Yeah, I noticed... :-)


> > There will never be "output EventQueues", since you're always
> > sending directly to the connected inputs, as far as plugins are
> > concerned. Whether you're actually sending directly to another
> > plugin, or to a host managed sort/merge proxy queue is pretty
> > irrelevant to the API and plugins, since it's a host
> > implementation thing.
>
> Well, there is an output-queue, it just might be someone else's
> input queue.

Yes, but I think it's more confusing than helpful to drag that in. 
What you *own* is your input Queues. When it comes to outputs, the 
only objects you own are these "XAP_target" structs.


[...normalized and audio...]
> I've always heard it called normailzed, and it jives with what I
> know.
[...]

Yeah, I know - but it's rather confusing when "normalized" in relation 
to *controls* means that out of range values are illegal.


[...]
> > And connecting an audio port means handing it a buffer pointer,
> > among other things? That is, buffer pointers aren't changed
> > frequently, and never silently.
> >
> > I *think* this should do, since it does not conflict with reuse
> > of buffers. Hosts can just figure out optimal buffer usage
> > patters when building nets. Right?
>
> That is how I have envisioned it - does this need special
> annotation?

Not really. Since we're using function calls (rather than shared 
variables), it's rather obvious. I was thinking out loud, and only 
came to the conclusion that this should work. :-)


[...]
> > I'm afraid this won't work for all hosts. There are situations
> > where you might want to run a plugin under a fake copy of the
> > host, in another thread, to perform non RRT safe operations
> > without disrupting the rest of the network.
[...]
>
> How does this apply to the host struct, though?  Do we need to
> change the host struct?  hmmm

The host struct is the plugin's interface to various host provided 
resources. Unfortunately, making these resources thread-safe would 
result in a major performance hit. (Consider the event system, for 
example.)

The easy way to deal with this is to just give plugins another host 
struct when you want to mess with them in another thread. The plugins 
won't be able to tell the difference, unless you want them to.

Of course, you'll have to disconnect any inputs before you start 
messing with that plugin, and if you're going to do something that 
makes the plugin send events, you have to make sure they're not sent 
directly to live Queues in the RT thread.

In short, just detach the plugin from any RT resources, and if you 
need to, hook it up with fakes. Do what you need, and then throw the 
plugin back into the RT network. (Or destroy it, or whatever you're 
up to.)


[...silence<...]
> We can track silence per Port, or just per Plugin (simpler, I
> think).

Simple, but insufficient for any but the simplest plugins. If you have 
multiple outputs, they can be completely independent. Your average 
multichannel/multitimbral synth without integrated mixer will have 
one or more outputs per channel, and you will most likely not have 
output from all of them at all times.


> Tail size is a good indicator - if we know that the plugin before
> the current plugin output silence, then we need only wait for the
> tail on this plugin to end and then it will be silent.  This relies
> on plugins getting their tail-sizes correct. Tail can be 0, finite,
> infinite.  Infinite tails always need processing

Yeah... Although I'm getting the feeling that it's more work for both 
hosts and plugins, than basically just "marking" buffers.

Speaking of which, the new Audiality mixer just has a flag to go with 
each bus accumulation buffer. Instead of clearing the buffers, I just 
set the flag to "silent". The first "unit" (mixer stage or synth 
voice) that hits a silent buffer gets to initialize it and set the 
flag to "in use". Whenever a mixer stage reads from a buffer, it 
checks the flag first, and then decides what to do - which in many 
cases means return instantly from process(). If you're doing 
something that results in no output (ie you're mixing silence into a 
buffer), just don't touch your output buffer. If it's marked as 
silent, it just stays that way.

The disadvantage is that everyone with an output has to be aware of 
this, because a buffer marked as silent is intended to be *ignored*. 
That is, it may (and generally will) contain garbage, which you'll 
have to clear out first, unless you have a replacing version of your 
inner loop.


> We can also do a call and answer style - the host tells the plugin
> that it's feed is silent.  When the plugin has died down and is
> also silent, it responds to the host that it is silent.

I like to think of "silent" as a property of the data. A boolean per 
buffer, basically. (Wherever we'd put that...) For inputs as well as 
outputs.

The FX plugins of Audiality get NULL pointers for silent input, but 
they use their state (multiple RUNNING states... *heh*) to tell the 
host whether or not they generated output for each block. The NULL 
pointers for inputs is nice and easy, and easy enough for plugins to 
fake themselves: Just point any NULL inputs at the host provided 
silent buffer. The output part "works", but only because FX plugins 
can only have one stereo in and one stereo out.


> Then the
> host can hot the bypass switch (is bypass optional?) or remove it
> from the net, or just stop processing it.

No, that won't work. A plugin being silent doesn't mean it's passive, 
or even that it won't start generating output again in a moment, even 
if there's no new input. A long feed-back delay fed with a short 
sound would intermittently generate non-silent buffers for a good 
while, and plugins following it could still have use for the "silence 
flag".

Besides, you cannot safely stop calling a connected plugin. If it has 
inputs directly connected to outputs of other plugins, there's no way 
you can know when to start calling the plugin again, since you won't 
even know about the events sent to it's Queues. What you get is a 
dead plugin draining your event pool. :-)


[...master and channel "queues"...]
> I didn't mean that they are separate queues - the host will ask the
> plugin for an event target for the master and each channel.  It may
> be the same Queue.

Ok.


> Should we define Event Target as a term and use
> it?

Well, it's an actual struct in Audiality. :-)


>   * EventTarget:
>         A tuple of an EventQueue and a cookie.  Plugins have an
> EventTarget for each Control, as well as a master and per-Channel
> EventTarget.

Not quite. Targets aren't really "input objects" in the API, but 
rather two other things:

	* The physical struct that you get filled in or
	  returned when you ask a plugin for one of it's
	  input controls.

	* An abstract object inside a plugin, representing
	  a control output.


In practical terms, something like:

	/* Get target for plug2 input control X */
	XAP_target target = plug2->get_control_input(X);

	/* Connect plug1 output control Y to that. */
	plug1->connect_control_output(Y, target);


and perhaps:

	/* Notify plug2 that it's input X is connected. */
	plug2->connect_control_input(X);


Plugins will most probably keep these in arrays or other structures 
internally, for their output controls, but the get_control_input() 
call will nearly always construct targets on the fly. All it does is 
pick the right Queue and transform the "full adress" of the control 
into something handy for internal decoding; ie a cookie.


> > >   The Host queries the Plugin for EventQueues via the
> > > get_input_queue() method.  In order to allow sharing of an
> >
> > I would prefer calling it get_control_target() or something,
> > given that what you actually get is not a Queue, but rather an
> > "atom" containing a Queue pointer and a cookie. Each of the two
> > fields of this "atom" (XAP_target) is completely useless on it's
> > own, but together, they form the complete address of an input
> > Control.
> >
> > >   Controls may also output Events.  The Host will set up output
> > > Controls with the set_output_queue() method.
> >
> > How about connect_control_output() or similar? Again because
> > queues have very little relevance on this level, and because
> > "connect" is
>
> Adjusted to get/set_event_target().  Good?

Still feels wrong... I have another idea, inspired by what's actually 
going on, as well as this "do we notify inputs after connection?" 
thing:

typedef struct
{
	XAP_queue	*queue;
	unsigned int	cookie;
} XAP_target;

typedef struct
{
	unsigned	class;
	unsigned	bay;
	unsigned	channel;
	unsigned	index;
} XAP_address;

Inputs:
	int open_control_input(XAP_address *a, XAP_target);
	void close_control_input(XAP_address *a);

Outputs:
	int connect_control_output(XAP_address *a, XAP_target *t);
	void disconnect_control_output(XAP_address *a);


Usage:
	XAP_address output = {...};
	XAP_address input = {...};
	XAP_target target;

	/* Connect */
	if(p2->open_control_input(&input, &target) < 0)
		return ERROR;

	if(p1->connect_control_output(&output, &target) < 0)
	{
		p2->close_control_input(&input);
		return ERROR;
	}
	return OK;

	/* Disconnect */
	p1->disconnect_control_output(&output);
	p2->close_control_input(&input);

("this" pointers left out for brevity.)


Looong names those, though... I don't want "event" in them, because 
events are not equivalent with controls; they're just used to operate 
them. Leaving out "control" might cause confusion with audio stuff, 
and dropping the "input" and "output" part would be generally 
confusing.


> > Another thought: Would plugins want to know when their inputs get
> > connected and disconnected?
>
> That is why these are methods - a guaranteed trap for connecting
> ins and outs, just like Ports..

Yeah, but I was thinking about the difference between asking for a 
target from an input and actually connecting it. As you can see 
above, this design makes things quite asymetric, but after throwing 
the close_control_input() counter part in, it looks rather nice, 
IMHO. The point is that both plugins now know when connections are 
made and broken. Without the close_control_input() call, receivers 
would be left in limbo if a connection fails in 
connect_control_output().


[...]
> > More importantly, when do you stop processing events? There are
> > two alternatives:
> >
> > 	1) Process only events that affect audio frames that
> > 	   you're supposed to process during this block.
>
> Which (not obviously) means 0 frames = no work.

Yes - but from a practical POV, why call run() in that case?


> > 	2) Process as many events you can, without skipping
> > 	   into the next block.
>
> Process all events up to and including 'now'.  For a 0 frame run(),
> all timestamps are 'now'.

No, timestamps are still valid, so there's no way to guarantee this. 
All events in the queue that match the current time will be 
processed, but any future events will be left in the queue.

Note that you get this behavior for free with a normal event/audio 
compound loop; it doesn't require any special cases or anything. (In 
fact, *guaranteeing* that there always is a "tail" event beyond the 
end of the block would eliminate the test for ev->next == NULL when 
readin events.)


>  The Plugin must process all events for
> now.

Yes.


> > >   Plugins can assume that VVIDs are at least global per-Plugin.
> > > The host will not activate the same VVID in different Channels
> > > at the same time.
> >
> > Well, as usual, I'd just like to whine some about it being
> > assumed that the sequencer is an integral part of the host. ;-)
> >
> > It could be an event processor or a phrase sequencer you're
> > listening to. Not that you'd care when you're a plugin in the
> > receiving end, but things get weird if you're about to implement
> > a plugin that outputs synth control events, while in the API
> > docs, it appears that only hosts can do such things...
>
> ok, ok - I'll reword it.  How should it be worded?  Assume an
> arpegiator or something that outputs VOICE.  Where does it get
> VVIDs?

Host call, just like allocating events and whatnot.


> Does it have to ask the host to reserve a bunch for it? 
> How?

>From the VVID prototype for Audiality:

/*--------------------------------------------------------------------
	VVID API
--------------------------------------------------------------------*/

typedef Uint32 ADY_vvid;	/* The reference */
typedef	Uint32 ADY_vvid_entry;	/* The object (for receivers only!) */

/*
 * Allocate 'count' Virtual Voice IDs.
 * The return value is the index of the first VVID in the host's
 * array, or a negative value if an error occurred.
 */
ADY_vvid ady_vvid_alloc(int count);

/*
 * Free a range of VVIDs previously allocated with ady_vvid_alloc().
 */
void ady_vvid_free(ADY_vvid first);

/*
 * Get the base address of the VVID table.
 *
 * IMPORTANT:
 *	This may *change* between engine cycles! It is therefore
 *	very important that plugins do not keep their own copies
 *	of the VVID table base address across process() calls.
 */
ADY_vvid_entry *ady_vvid_table(void);


Of course, in XAP they'd be function pointers in the host struct 
instead. (Audiality's "host struct" is still an implicit singleton - 
or "global stuff spread all over the place", if you like. :-)

ady_vvid_table() would probably be better off as a pointer in the 
struct. (Why have a function call? Can't remember why I have one 
here...)


> Good feedback - looking for more - specifically, help me flush out
> the Table of Contents so all the details get in there...

Well, I'm too tired to deal with structured things like that now, so 
I'll have to have another look at it tomorrow... ;-)


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
   --- http://olofson.net --- http://www.reologica.se ---