[linux-audio-dev] Plugin APIs (again)

David Olofson david at olofson.net
Tue Dec 10 13:00:00 UTC 2002


On Tuesday 10 December 2002 07.17, Tim Hockin wrote:
> > > a Channel has p Controls and q Ports
> >
> > Well, a Channel can have p Controls OR p Audio Ports. I would say
> > that a Channel can vave p *Slots* - where a slot can be one of:
> > 	Audio Input Slot
> > 	Audio Output Slot
> > 	Control Input
> > 	Control Output
>
> Ok - I see how you've split this out, but why?

Because then you only have to worry about types and styff for the 
*Bay* not for one or more Channels.


> What is the
> reasoning for having channels only containing one type of thing? 

Fewer dimensions of indexing when referring to things. In short, I'd 
prefer as much type and hint info as possible to be concentrated in 
as few places as possible.

Here, basically *everything* is in the Bay level. The rest is just 
about saying how many Channels you want each Bay to have. Channels 
might have names, but not even that is required; you only really need 
to have names, type, hints and stuff on Bays.


> And besides that, can't Event Ports be bi-directional?

No. An Event Port is basically just a queue with both head and tail 
pointers. You can add events at the tail, and get events from the 
head; that's how communication is done normally. The physical Event 
Port object is what you refer to whether you're writing to it or 
reading from it.

Event Ports that are owned by the Plugin are used for input, period. 
(Well, you could use them internally in whatever way you like, but 
that's another story entirely.)

Control Output is not done through ports, but rather something I'd 
call a Control Output Slot, which is probably better thought of as an 
abstract object from the plugin POV. Internally in a plugin, it would 
loos omething like this:

	struct MY_control_output_slot
	{
		XAP_event_port	*port;
		Uint32		control_id;
	};

When you're told to connect an output, you'll get

	* A Bay:Channel:Control address of the output control
	  to connect.
	* A pointer to the Event Port to send related events to.
	* A control ID to mark events you send with, so the
	  receiver knows what you're talking about. (That ID
	  generated by the receiving plugin, remember?)

>From now on, you're expected to send any events related to that 
Control Output of yours to the specified port, marking them with the 
specified control ID in the "index" field. (Or maybe we should call 
it "cid" or something? It may not be an index at all, and either way, 
senders should never try to understand the values.)


> > Event Ports don't really belong in this picture. They're like a
>
> I later read what you mentioned about get_event_port() and it was
> nicely abstracted.  Having the plugin pass a cookie to the host to
> be passed back on the port was my first thought when I saw that
> idea.  So ignore what I've said about event ports for the duration
> of this discussion.  The only way a plugin exposes it's event ports
> is through some abstracted method.

Yes. I think that's as simple and yet flexible it gets.


> > Bays are actually a way of grouping channels of the same kind
> > together. I initially called them "Channel Kinds", until I
> > realized that I had already "invented" the term "Bay" for exactly
> > this thing for MAIA.
>
> I'm not conviced Bay has the correct connotation...

Well, the intention is that it should be thought of something like a 
physical "panel" or area on a real device, where you have a number of 
jacks, all of the same kind.

For example, my Layla has one bunch of analog inputs, and one bunch 
of analog outputs. In MAIA, the inputs would go in one Bay, and the 
outputs in another.

Maybe there's a better word for it.

Simply "Group"? No, "Group" makes me think of something *much* more 
lightweight; more like something you would have on the UI level to 
keep physically unrelated (from the API POV) channels together.


> > >  This is what I was
> > > thinking about as I slept (weird dreams...)
> >
> > You have those too? :-)
>
> With all the discussion on this, and the two of you running much
> faster than I can reconcile all the details, yeah.  I'm having
> racing-mind syndrome.
>
> > > Plugin {
> > > 	/* many plugins will have 1 template, but they can have more.
> > > 	 * multi-timbral instruments (synth part, drum part, ...) and
> > > mixer * slots (mono, stereo, ...) come to mind
> > > 	 */
> > > 	n_channel_templates;
> > > 	channel_templates;
> >
> > This sounds pretty much like my Bays. You have one template for
> > each "kind" of channel you want, and then you can have varying
> > numbers of channels of each kind?
>
> Yes.  The difference is that I don't see why splitting out Controls
> is useful.

Controls are on a different level, and slightly more lightweight than 
Channels. Channels are physical objects, while Controls (from the API 
POV) exist only as indices or control IDs.

So, with Audio, you have one Channel that handles exactly one mono 
audio stream, and that's it.

With Controls, you have one Channel that has an array of Controls - 
but not even the *Channel* must have an Event Port. The closest you 
ever get to a Control of a Plugin is the control ID (or cookie) that 
you get when you want to connect to it.


>  The way I envision a Channel in my mind is a 'Page' of
> the plugin - specifically instruments.

That could be useful - but I also think it's rather restrictive, and 
more importantly, that it doesn't belong in this level of the Plugin 
API. Expressing these relations is better done through some kind of 
lightweight hinting, IMHO.


> I didn't see any reason for
> channels in effects

A stereo in-stereo out effect would have two input Channels (one Bay) 
and two Output Channels (another Bay), and at least one Control Input 
Channel with a bunch of Controls (a third Bay), the way I see it.


> until mixers were brought into it, and I didn't
> see any reason for different controls per channel (even now, I'm
> not sure - can you convince me further?).

There are not different controls per Channel, but rather, you may 
have more than one Bay of Control Input Channels - and then, each Bay 
may have it's own set of Controls, since the Bays are completely 
independent objects on the API level.

Anyway, looking at most h/w synths, they have a number of channels, 
each controlled through it's own MIDI channel. Then there's a mixer 
section - but oops - there are no MIDI channels for this, so the 
mixer can't be controlled using normal MIDI CCs! Instead, most synths 
publish the mixer parameters only through custom SysEx.

Now, comparing by idea to MIDI in this context, Bays would be like 
(physical) MIDI ports. Adding a separate Bay for mixer controls would 
be like adding an extra MIDI port on the synth, meant specifically 
for mixer control and automation.


Let's compare that to the crude hack I'm currently using to let MIDI 
files control the mixer section. I've simply stolen a range of CCs 
from each MIDI channel, which are routed to a mixer channel instead 
of a synth channel. So, you can control 16 synth channels and 16 
mixer channels. However, as of now, there are only 8 mixer 
channels... So what happens if you send mixer events on the other 8 
MIDI channels? (Nothing.) How nice and clean is this? (Not very, 
IMHO.)

I'd rather not have a brand new plugin API that forces me to do the 
same thing.


>  Sometimes giving up a
> bit of flexibility makes things much easier.

Yes. However, I don't see what is made simpler with the Template 
approach. I'm probably missing some points, since I can't even tell 
for sure whether your Templates do provide the same number of 
dimensions of indexing or not.


> Each Channel (page) has similar or identical controls - LFO,
> Filter, whatever as well as similar or identical voice controls
> (VELOCITY, PITCH, BALANCE).  Controls are identified by {channel,
> index} tuples.  I don't see any reason to put voice controls in a
> separate channel from channel controls.

The reason is that the host would just see two normal Bays of Control 
Channels. One just happens to be hinted "these are Voice Controls for 
that other Bay", in case some higher levels of the host (or the user) 
cares.

The point is that you would only need to implement "get info on 
Controls for Bay N" in the host, and for connections "connect to 
Control X, Channel Y, Bay Z". No special cases for Voice Controls.

And if someone comes up with a third kind of Controls (whatever that 
would be), they won't need explicit API or host support either. If 
you don't care about Voice Controls at all, you don't even have to 
know they exist.


> > Still a 1D array of channels, that is?
>
> This is a minor issue by comparison :)  Whether we make it an
> unsorted 1D array, a sorted 1D array, or a multi-dimensional array
> is really nothing but a matter of indexing.  We can argue this
> after :)
>
> > I'm thinking about Bays as an array of objects that each has an
> > array of Channels - but it doesn't have to be that way
> > physically,
>
> I take back my last comment - if that is what a bay is, it is
> really just another way of saying 'channels are grouped with their
> own kind' and you are suggesting using it as an index into a 2D
> array.  Instead of {channel, index} tuples as above, you'd have
> {Template, Channel, Index} which is exactly what you've proposed
> with different words.

Yes, I think so.

It's about the same thing as with the Event Ports; if you ask for 
Bay:Channel (audio), or Bay:Channel:Slot (controls), or whatever and 
the plugin just gives you "something" to connect to, and perhaps a 
cookie to mark data with, the Plugin can arrange things in any way it 
wants physically.


> > > Eventport {
> > > 	label;
> > > 	name;
> > > 	/* anything else? */
> > > }
> >
> > Mine look like this:
>
> So do mine - I was suggesting an eventport descriptor, for hints. 
> But since that can be completely opaque to the host, it doesn't
> matter.

I see. (I realized later that the struct you proposed simply doesn't 
exist when you don't have Event Ports as visible objects in the API.)


> How does the host know about ports that controller plugins intend
> for other plugins?

There is no such thing. When the host wants to connect a Control 
output on plugin A to a Control input on plugin B, it'll have to ask 
plugin B about the port and control ID of that control, and then tell 
plugin A to connect to that.


> Assuming the arpeggiator, which outputs
> voice_on (or whatever!) events - how does it tell the host 'I have
> a special port which is control data?

It doesn't. There are no output ports, just Event Output Slots, as 
described above. The plugin just tells the host about their existence 
by mentioning a Bay of Control Output Channels, and then the host 
will assume that the plugin knows what to do when it's told to 
connect these outputs to various places.


> > Well, it just happens that channel + voice control is a very
> > practical model for most synths. It's probably not perfect and
> > without restrictions, but I think we can do a lot of cool stuff
> > before seeing much of these issues.
>
> I also think that we can't accommodate EVERY whim of plugin
> writers. They will have to make some adaptations of their perfect
> model to fit the plugin architecture.  What we are doing RIGHT NOW
> is trying to minimize it.  We must acknowledge that we can't give
> them everything they will want, but we can give them enough to
> accomplish their goals without asking them to be too flexible.  :)

Exactly. We should try to avoid building more implied structure into 
the API than required, since that adds restrictions as well as 
complexity. As few different kinds of objects as possible; 
"everything is a..." (within reasonable limits) etc.


> > 	1. Have two control ranges for Event Input Channels.
>
> They don't need to be seperated, really.  All the host knows of
> them is their descriptors and indices.  Now, that said, there isn't
> any real reason NOT to split them, either.

There's just one problem: Voice Controls need an extra argument 
(VVID) that is irreleavant to Channel Controls.

You can hint Controls as Voice Controls, mark them as different by 
using a different Channel Type or whatever - but fact remains: They 
Are Different. Voice Controls need a slightly different protocol. 
(Either different events, or use of an argument that's ignored by 
Channel Controls.) Weird things (or nothing at all) may happen if you 
connect Channel Control outputs to Voice Control inputs or vice versa.


> > 	2. Allow only one range of controls per Event Input Channel,
> > 	   and instead, use an extra bay of Voice Control Channels, if
> > 	   you want Voice controls. One could mark the Bays with hints
> > 	   suggesting that they are closely related.
> >
> > Now that the strict physical relation between channels and Event
> > Ports is eliminated, this doesn't matter at all performance wise,
> > but it might make the API a bit cleaner.
>
> I think this is over-normalizing it.  Some Channels really want to
> have their own i/o ports to make the paradigm more comprehensible.

A Channel *is* an "I/O port" the way I think of it. A Control Channel 
may have multiple Controls, which is actually in conflict with my 
philosophy - but then again, Controls are not physical objects on the 
API level.

Hmm... One might say that if Bays are horizontal, Templates are 
vertical. Since you can have multiple arrays of different types of 
channels inside a Template, one could say that a Template contains a 
bunch of something that resembles Bays.

So, how about using the "Bays or Channels" system in the low level 
API, and then provide (optional?) information that provides the 
grouping information of the Templates?

If a plugin provides no Templates, Bays are to be considered 
completely independent, except for the only hard relation I can see 
that they might need: Linked Channel counts. Hosts may ignore 
Templates entirely, and just have users go by Bay naming to figure 
out the relations. (Not too hard if Bays are named "Strip Inputs", 
"Strip Insert Outs", "Strip Insert Ins", "Strip Controls", "Bus 
Controls", "Bus Inserts" and things like that.)


> > > 	n_inports;
> > > 	n_outports;
> > > }
> >
> > I'd rather think of Bays as being in/out neutral, and use
> > different Channel types for input and output. Only one array of
> > Bays needed that way.
>
> Why?  Why convolute the host and the plugin by splitting them, if
> they are naturally one entity.

They are not, in any way I can think of. You may have only Control 
inputs and Audio outputs, or vice versa, and nothing is both an input 
and an output.


>  Again with mixer slots.  Mono slots
> have one input and output to the mixer's bus. Now, I can't think
> of anything but mixers for which that really makes any sense. 

That's because it's just a special case. Many real mixers indeed have 
insert jacks that are both inputs and outputs (stereo 6mm where one 
"channel" is used as the return), but that just means you'll need 
special cables to connect anything to it. (Anything with normal, 
separate inputs and outputs, that is, which goes for the vast 
majority of devices AFAIK.)


> Perhaps a 'string-section physical model'.  You add a
> 'CELLO_PLAYER' Channel and a 'VIOLIN_PLAYER' Channel. :)

How does that apply here?


> I want to keep this all simple for the simple synths, too.

So do I. All you have to say is this:

	Bay 0:
		Type:		INPUTS, CONTROL, CHANNEL
		Min Channels:	1
		Max Channels:	1
		Granularity:	1
		Linked w/ Bay:	-1
		Channel Names:	{"Control Input"}
		Info:		<list of control names + hints>

	Bay 1:
		Type:		INPUTS, CONTROL, VOICE
		Min Channels:	1
		Max Channels:	1
		Granularity:	1
		Linked w/ Bay:	0
		Channel Names:	{"Voice Control Input"}
		Info:		<list of control names + hints>

	Bay 2:
		Type:		OUTPUTS, AUDIO, FLOAT32
		Min Channels:	2
		Max Channels:	2
		Granularity:	2
		Linked w/ Bay:	0
		Channel Names:	{"Left Output", "Right Output"}
		Info:		<>

...and you have a stereo synth with one channel; both channel and 
voice controls.

The host will eventually give you an array with two audio buffer 
pointers (saying it's for Bay 2, but you don't have anywhere else to 
put it) and then it might start to ask you for Event Port pointers 
and cookies for the controls you've listed for Bay 0 and Bay 1. Just 
give it your only event port, and return index as is for cookie. You 
may ignore the "channel" and "bay" arguments, since different events 
are used for channel and voice control.


>  Maybe
> all inputs/outputs are on the master and are fixed.

Most synths will have only outputs in the master section. Why even 
mention inputs if you don't have any? With Bays, you don't have to 
fill in zeroes in a bunch of fields for things you don't want - you 
just don't export any Bays for that stuff.

Besides, Bays + Channel Type enum means you can add Channel Types to 
the API without breaking binary or source compatibility with older 
plugins. You might even be able to use *new* plugins in old hosts; 
you just won't be able to use Bays of unknown types.


>  You can add
> banks of controls and parts by adding Channels, but the number of
> in/outs is fixed?

Why would they be different? Channels include control channels as 
well as audio channels in my design.


> I don;t really see why it need be.  Maybe both. 
> Master can have in/outs and channels can have in/outs.  The host is
> responsible for realizing 'I just added a channel with 2 outputs,
> I'd better connect them to buffers'.

Sure. And this should not be a major issue.

Adding new Bays on the fly comes with a bunch of interesting 
problems, though - but it might not be *that* much harder. It's just 
that it doesn't mix well with the "host calls some plugin function to 
get a list of Bays during load time" design. Adding Bays would be 
something that a plugin *instance* does spontaneously through a host 
callback, as a result of a control change or something.


> (Of course this is non-RT
> behavior in hard RT systems).

That's not a nice assumption to make, IMHO. Hosts and plugins should 
at least not be explicitily *prevented* from doing this stuff in real 
time. (This is another reason to proide some form of generic "call 
this function from some other thread" feature in the host. Standard 
hosts would just do it in the RT thread and take the risk of a 
drop-out, but more sophisticated implementations would be possible.)


> Simple synths have all their controls and outputs in a single
> channel (or simpler in the master).  Drum samplers can have as many
> channels as they want, each with 1 output and whatever controls. 
> Mixers can have stereo outs on the master and any number of mono
> channels.  That seems pretty powerful and pretty simple, to me.

Ok, but I think there's a lot of "I don't want this and I don't want 
that..." filling in to do, and also hard restrictions built into the 
API. Small, simple structs and enums allow you to extend things later 
on without breaking minary compatibility, and it also allows plugins 
to *completely* ignore things they don't care about.


> > Linking Bays WRT Channel count may be useful, though... (If you
> > have a mixer, you'd probably want exactly one strip automation
> > control input for each stip audio input. :-)
>
> Or just make them together since they are really one logical
> entity.

I don't think they are. This is just a special case that applies to 
mixers and some other things.


> As an old database teacher once told me - always be
> suspicious of mandatory 1-1 relationships.

Exactly. That's why I don't even want to *suggest* anything about 1:1 
relationships on this level. :-)

The only required feature I can see is this "link" thing, which you 
only use when two or mora Bays actually *have* 1:1 relations.


> Basically we're in agreement except on terms and whether a
> Template/Kind/Bay is really single-typed or not.

Yes. I'm suggesting that we use as small and simple objects as 
possible, and have one for each type of channel we actually want. 
What I don't like about your design is basically that the set of 
supported connection types (audio and control) is hardcoded into the 
Template struct, rather than available as an enumeration or similar.


> This model is shaping up nicely - I'm so glad to have excellent
> feedback.

Yeah, I think we're getting somewhere here. Some real hard problems 
have been solved in rather nice ways, and it seems like we're mostly 
arguing about cosmetic details and how to balance simplicity and 
flexibility.


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
|    The Multimedia Application Integration Architecture    |
`----------------------------> http://www.linuxdj.com/maia -'
   --- http://olofson.net --- http://www.reologica.se ---



More information about the Linux-audio-dev mailing list