[linux-audio-dev] Plugin APIs (again)

David Olofson david at olofson.net
Thu Dec 12 05:38:00 UTC 2002


On Thursday 12 December 2002 09.28, Tim Hockin wrote:
> > > So _every_ audio 'Channel' has one i/o port?  Assuming I buy
> > > this whole model, it seems like multiple I/O (of one flavor - I
> > > or O) is right.
> >
> > You could say the Channel *is* the port (either I or O) for
> > audio, whereas for Controls, it's more like MIDI and CCs; one
> > Channel has an array of Controls.
>
> But why does it have to be that each Channel (using your terms
> until we find better ones) is a single I/O?  It seems like that is
> a lot of meta-data and indirection to just get a single mono port. 

An array of them actually. Oh, well... My original terminology was 
indeed confusing, so I would like to mention that Channels are 
abstract objects, while these things that can have only one I/O are 
*Ports*. One Channel may have multiple Ports of various kinds. 
Question is how to describe them to the host and users.

I think there are better ways, though.


> You mention further down about having a Bay for "Left Audio" and
> "Right Audio".  There is an implicit 1-1 mapping there that is not
> explicity defined anywhere and is pointless.  If you want them
> bonded, make the Bay be a stereo bay.

How would you do that - and how would anyone know that Ports belong 
together in pairs, one pair per Channel?

Then we'll need that granularity thing again... We're just pushing 
the problem around here.


[...Total Control chassis...]
> It is somewhat elegant.  The only ugly bit is that the Plugin
> writer needs to define a compatibility matrix somehow.

Yes... No matter how we keep turning things around and inside out - I 
don't think there is a single structure that fits all, without having 
a rather large number of indexing dimensions. All "sensible" 
solutions are only sensible for a few kinds of plugins. *heh*


> Now, I'll accept your assertion that Channel Control and Voice
> Control are different Bays.

And I'm beginning to think that there is no point in doing that. :-)


>  I don't see ANY reason to have I/O be
> mono.

What is a stereo I/O? How is it described, and how do you index them 
when making connections?

This was what granularity was about; specifying that one Channel 
corresponds to a certain number of Ports in this Bay, so you can have 
stereo, 5.1 or whatever. Physically, the ports are just one long 1D 
array, but the granularity field bundles them together in a "soft" 
structure that may be of use to hosts and users. It also makes 
linking of Bay Channel counts useable, since you link *Channel* 
counts, rather than Port counts.

So, if the synth has one Control Port for each set of 5.1 (6 audio 
Ports), you get one Control Port (in the Control Bay) and 6 audio 
Ports (in the Audio Bay) for each Channel.

I'm not all that happy about this any more, though. Read on.


> Now that all that is out, I'm still not convinced that splitting my
> Channel struct { I/O, channel control, voice control } is needed.

Nor am I...


> > > > The reason is that the host would just see two normal Bays of
> > > > Control Channels. One just happens to be hinted "these are
> > > > Voice Controls for that other Bay", in case some higher
> > > > levels of the host (or the user) cares.
> > >
> > > I think "of course they care!".  You've defined an implicit 1-1
> > > mapping of voice-control channels to channel-control channels.
> >
> > Well, how explicit does it have to be? Isn't the fact that the
> > very meaning of the link feature is to tell the host (and
> > everyone else) that there *is* a 1:1 mapping sufficient? Then,
> > how is grouping them in a struct used during initialization?
>
> The 'link feature' is ugly, IMHO :)

Agreed! :-)


> > *heh* Yeah, Ports would be better. (And these Event Ports should
> > be renamed Event Queues or something.)
>
> heh - I did that below :)

Well, I did it in the headers as well. Now, I just call them Queues - 
since there is only one kind of them in the system anyway.


> > Or we can use host callbacks, or build a queue of events to
> > describe the plugin. (The latter is what the tiny MAIA prototype
> > did.) More flexible, but may not look very clean... One advantage
> > is that if you forget to send something, the host will notice
> > that it never came in, so you won't have garbage or
> > non-intentional NULLs without knowing about it.
>
> This is pretty much what GStreamer is doing.  I don't like a ton of
> 'register me' functions.  Ideally a simple descriptor for a simple
> plugin can be static data, and all the plugin does is return a
> pointer to it, or similar SIMPLE.

Yes. It's just that we tend to get too many dimensions whatever we 
try...


> > It could work for custom audio formats, custom events and that
> > kind of stuff, but even then, you'd need host support for
> > managing buffers of custom sizes. (Interleaved multichannel for
> > inter-plugin busses and that kind of stuff.)
> >
> > We'll see when we've dealt with the *basic* issues. :-)
>
> yeah - I raised it as a joke, but it's not really :)

Well, I already have "User Events" in the headers... I'm not sure I 
want to think about the implications of that right now, though! :-)

Either way, if we can make sense of this Bay/Channel mess, I suspect 
that at least multiple audio datatypes will fit in rather nicely. 
It's kind of like the note/linear pitch thing; equal type IDs fit, 
different IDs do not, and need conversion.

So, if everyone just uses float32, there's basically no difference 
from having *only* float32. Port types will have to be compared 
before connections are made anyway.


> > > > Exactly. That's why I don't even want to *suggest* anything
> > > > about 1:1 relationships on this level. :-)
> > >
> > > But this is full of them. A mixer's slot-controls bay is
> > > useless without a corresponding audio-in.
> >
> > But that's something decided by the plugin author - not by the
> > API.
>
> I think that almost every plugin will have a number of 1:1
> releationships. Every synth will have {1 Channel Control, 1 Voice
> Control} associations.

Yes. Some will have multiple *kinds* of 1:1 relations. (Mixer strips 
and bus strips.)


> > Channels. That's what the link field means; "I must have the same
> > number of Channels as Bay X." (I didn't say that anywhere?
> > Oops...)
>
> An icky way of describing associations, IMHO.

Yeah. "Soft" description of a hard relation.


> > Yeah. There are too many objects in this design, or too many of
> > them have inappropriate names... *heh*
>
> Well YOU are the one who wants to add Bays, which forces me to
> counter with Templates :)

So, if we remove both, we can get away with only Channels and Ports? 
;-)


> > Here's another confusion. (Probably caused by me not describing
> > what "linked bays" actually mean.) A Bay would not have controls,
> > but each one of it's Channels would.
>
> I mis-spoke, that's what I meant.

Ok.


> > Channels are indeed a lot like MIDI channels in this respect.
>
> Except for I/O Channels which are nothing more than an I/O port
> described by two separate entities.

Right.


> > > * Preset:
> > > 	A stored or loaded set of values for all Controls in a Plugin.
> > > 	//FIXME: can presets be per-Bay?  per Channel (group of bays)
> >
> > A preset should always be for the entire Plugin. A Bay is just a
>
> Good - that's what I thought, too.  If we assume MIDI style
> channels, though (multiple configurations of the same controls),
> then loading a preset into a channel might be very cool.

Well, yeah, considering that we're talking about multichannel synths, 
it would actually be pretty much a *required* feature. Didn't think 
of that, for some reason... (Probably just forgetting that the host 
really *is* supposed to handle *all* preset data.)


> > > * Event
> > > 	A time-stamped notification of some change of a Control.
> > > 	//FIXME: are events ONLY about controls?  What isn't?
> >
> > That's a good questing, actually - and it demonstrates this need
> > to deal with events that are *not* about controls. There will
> > most probably be a few of those, but frankly, I can't tell for
> > sure whether they're *really* needed or not.
>
> Right - even voice_on/off can be control(s)

Yes...


> > Anyway, no, events can be about other things as well.
>
> Example?

Connections, possibly. Since a plugin just gets a pointer and a 
cookie, there are no RT issues with this - and so the host may as 
well feed the data to plugins through events, instead of function 
calls. Sample accurate timing for (dis)connections - whatever that 
might be useful for.

Tempo changes?

And these scary User Events, should we decide to allow them at all.

Can't think of much, really. Most things are best described as 
Controls - and those are manipulated using a small set of standard 
Events.


[...]
> > thing. That way, a Port is just something you use to make a
> > connection to a Channel - regardless of whether we're talking
> > about audio or controls.
>
> Fine - Audio Port and Control Port.  And what lives in a Control
> Port is an Event Queue.

Or a bunch of them - or you see one that's shared by multiple Control 
Ports. Doesn't matter, as you're simply supposed to send Events to 
whatever Queue the plugin tells you to use. :-)


> > A Channel would then refer *only* to an abstract object inside
> > the plugin.
>
> uggh.  For the time being let's take what you've been calling a
> Channel (an instance of what a Bay describes) and call it a Foo. 
> Now, we can agree that a Channel is a set of Foos, probably in
> different Bays.  Totally abstract - a purely relational entity.

Yes...


> > But there's a problem with this as well: The Channels of a Plugin
> > don't *have* to be all of the same kind... For example, a Mixer
> > would
> >
> > So, what's the big deal? Well, nothing - just that it might be
> > rather confusing if there can be several "Channel 3" in the same
> > Plugin, that are in fact completely different things. One could
> > be mixer strip 3, while the other would be bus strip 3...
>
> And this is why Channels should be addressed one-dimensionally :) 
> A host really wants to instantiate Channels, usually.  By
> instantiating a Channel, it gets all the Foos, each in their own
> Bay, all associated together some how.  Channels are addressed
> linearly, and Channels of the same type are not necessarily grouped
> together.
>
> So why do we need Bays and Foos again, and not just
> ChannelDescriptors? Simpler is better ...

Well, the problem with addressing Channels 1D is that Channels move 
around if you change the number of Channels of a certain kind. So, 
you're pretty much forced to reinstantiate the plugin to change the 
Channel count.

Anyway, the term "Bay" needs to be redefined, I think.


> > > * EventQueue:
> > > 	A control input or output.  Plugins may internally have as
> > > many EventQueues as they deem necessary.  For each Control, the
> > > Host will ask the Plugin for the EventQueue on which to deliver
> > > Events.
> >
> > Yes - although referring to an EventQueue as an output might be
> > more confusing that helpful. A Plugin will *never* use an
> > EventQueue for output, but rather just send each event to
> > whatever EventQueue the host told it to send that kind of events.
>
> Hmm, I'd been debating whether it was simpler to have the Plugin
> write to a real EventQueue and have the host detach the Event list
> and re-attach it where it belongs.  It's a simple operation, and it
> removes the need for a host to conditionally shadow EventQueues. 

But it introduces the need for Events to contain full target 
addresses, and for the Host to dispatch the Events to the various 
target Ports they're meant for. Theoretically, every single output 
Control may be connected to it's very own Queue.


> It means a HAIR more work in the host, but less complexity. 

Nope. You kill the "automatic" Event routing that sending directly to 
Queues brings.


> However, then it needs to ALWAYS read ALL Control-Out ports on a
> Plugin.

That too. With direct sending, the only extra work ever done is for 
these shadow queues that are needed sometimes. Other than that, the 
only checking is the one that Plugins have to do all the time anyway.


> I guess a host that wants to can do that itself and just
> shadow EVERY port.

Yes, but why? (Except for studying the event flow into plugins, 
obviously. Hmm... There is an upcoming market for cool debugging 
plugins here. :-)


> Fine, you're right, probably.  Plugins only
> write Control-Output iff the Control port is connected.

Exactly. NULL Queue pointer means we ignore anything to do with that 
Control output. (Or if you have *lots* of Controls, just keep all 
connected Controls in a list, and use that as a task list. Plugin 
author's choice.)


> Does a
> Host need to disable_port() on the Control out ports, like it does
> audio?

It will have to explicitly tell the Plugin to disconnect a Control, 
before actually removing the target Queue or anything - or Bad 
Things(TM) will happen.

The point here is that I don't want to have the Host directly mess 
with pointers and stuff that the Plugins use.


I'm not even sure that should be allowed for Audio Ports... For MAIA, 
I had this idea about using events for changing audio buffer 
pointers. That way, you could change buffers in the middle of the 
process() call, and even have plugins deal directly with buffers of 
different sizes.

However, there is one major problem with that, which becomes obvious 
when you actually try to implement an event controlled plugin: It is 
no longer straightforward to use buffer pointers + indices in the 
inner loop. If you studied my example from Audiality, you might have 
noticed that there's an offset tracking the start of each "fragment". 
That offset would all of a sudden become invalid if buffer pointers 
are allowed to change mid-buffer!

So, we might as well do it fast and simple: Hand arrays of host 
controlled buffer pointers to Plugins.


> > So, instead, the stereo pairs could be expressed as two Bays,
> > linked together in Port Count, both with Audio Input Ports. One
> > Bay would be named "Left Inputs" and the other "Right Inputs".
>
> Or there is a StereoOut bay which has two Audio Ports.  I prefer
> this.

That's not my original idea of a Bay - but it's closer to what I have 
in mind now.


> Or Channels.  It's simple.  The terms are already understood and
> even map onto a well-known paradigm reasonably well.  The
> complexity goes down.  It's at least one less object in our model. 
> The ONLY things we lose:
>
> * Ability to instantiate an arbitrary number of Audio ports - bad
> anyway, since plugins will all have fixed or small ranges of
> allowable values.
>
> * Ability to ignore Inputs if you have none - now you need to
> explicity say '0 inputs'
>
> * Ability to later add new Bay types besides the currently known
> ones.  The only thing I see this as a loss on is Audio formats, and
> truthfully, I don;t care.
>
> Simple Channels can be expanded to:
>
> struct XAP_ctrl_channel_desc {
> 	char *name;
> 	char *label;
> 	unsigned flags;
> 	int n_controls;
> 	XAP_control *controls;
> };
>
> struct XAP_io_channel_desc {
> 	char *name;
> 	char *label;
> 	unsigned flags;
> 	int n_ports;
> 	XAP_port *ports;
> };
>
> struct XAP_Descriptor {
> 	...
> 	XAP_ctrl_channel_desc *ctrl_channel_descs;
> 	XAP_io_channel_desc *io_channel_descs;
> 	int **channel_compat; /* if (compat[ctrl][io]) */
> 	...
> 	int (*xap_new_channel)(int ctrl_idx, int io_idx);
> 	...
> };

Ok... A few questions on this:

	* Is a *_channel_desc able to express whether it's an
	  output or input?

	* What is the relation between control and I/O channels?

	* How would you handle say, ctrl in + one audio in +
	  two audio out?


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
.- M A I A -------------------------------------------------.
|    The Multimedia Application Integration Architecture    |
`----------------------------> http://www.linuxdj.com/maia -'
   --- http://olofson.net --- http://www.reologica.se ---



More information about the Linux-audio-dev mailing list