[linux-audio-dev] Plugin APIs (again)

Tim Hockin thockin at hockin.org
Tue Dec 3 17:04:00 UTC 2002


I'm replying to all the replies in one email, hope no one minds :)


> From: Steve Harris <S.W.Harris at ecs.soton.ac.uk>

> Fisrt off, I agree with Conrad, its better to make this fucused on, and
> really good at, one task (ie. instruments), that be OK at lots of things.

Agreed withone exception - It is meant to be good at instruments, effects
and outputs.  In my worldview almost everything is a plugin.

> > typedef struct OAPI_ctrl_int		OAPI_ctrl_int;
> > typedef struct OAPI_ctrl_float		OAPI_ctrl_float;
> > typedef struct OAPI_ctrl_enum		OAPI_ctrl_enum;
> 
> Enum's good, I'm not so sure about int or string, a well defined enum type
> probably makes string unneccesary and int is probably not needed.

Strings are for things like files, primarily.  There is no way to
autogenerate any sort of UI for anything dealing with files without some
understanding of strings natively.  Int is merely for completeness.  There
are things that are better represented as ints, and since there is this
notion of type-specific controls, it makes some things just a hair easier.
I had bool in there too, but realized bool is a special-case int.  Maybe int
is a special-case float, but I don't really think so.  Simple, but not too
simple.

> > /* Plugin types: all OAPI plugins are one or more of these. */
> > #define OAPI_TYPE_SOURCE		0x01
> > #define OAPI_TYPE_EFFECT		0x02
> > #define OAPI_TYPE_SINK			0x04
> 
> I dont think this helps, it should be obvious by inspection whether an
> instrument is appropraite for the situation the host wants to fill.

I think I agree with this - I left a fixme to that effect in the header..

> > 	/*
> > 	 * These are display-friendly fields, which hosts can display to
> > 	 * users.
> > 	 */
> > 	const char *oapi_version;
> > 	const char *oapi_author;
> > 	const char *oapi_copyright;
> > 	const char *oapi_license;
> > 	const char *oapi_url;
> > 	const char *oapi_notes;
> 
> If you're going to support metadata, support the Dublin Core, but generally
> extrnal metadata is better than internal. Just provide a unique ID and
> metadata can refer to that.

Firstly, I hate the idea that a plugin writer needs to procure a unique ID.
Secondly, I think some meta-data needs to be IN the plugin.  Now, maybe
'notes', and 'url' can be outside the plugin, but
author/copyright/license/version can't.  Think of commercial plugin (I know,
pipe dreams, but just pretend :).  As a commercial vendor you DO NOT want it
to be trivial for a person to 'lose' the info about the license.  I can
think of a several reasons to keep it all together, and no good reason NOT
to.  Specifically, this data is inseperable from the plugin.  As an author,
I want this data joined at the hip with the plugin.  Further, the idea that
a plugin consists of a single .so file appeals to me.  Presets?  I guess those
can live entirely in external files.  I can't see ANY good reason to make
such simple meta-data reside externally.  Maybe I can be convinced - a few
people chided in on this point - can anyone tell me WHY?

> > 	int oapi_inputs_min;
> > 	int oapi_inputs_max;
> > 	int oapi_outputs_min;
> > 	int oapi_outputs_max;
> 
> Variable numbers of i/o is interesting, but I'm not sure if it
> overcomplicates things from the instruments point of view. Can you explain
> why you think this is a good idea?

It stems from experiences with virtual drum machines and synths.  It is very
nice to provide a plugin that does things like slicing (thing .rex) or
drum-pads and route each pad/slice to a different set of FX.  Further, I
envision a mixer effect which has multiple inputs, and potentially multiple
outputs, and wants to process each input discretely before mixing.  I didn't
think it made sense to make a hard limit, much less a limit of 1.  The
reality is that MOST instruments/effects will have 1 in and/or 1 out.  Those
with more than 1 should be presented by the host as needing special
attention, and can make assumptions as the author desires.  For example, a
drum-machine with 16 outs which has only been connected on the first input
can assume that the other 15 pads should be internally routed to the first
output.

Perhaps it is useful to design some way for the host to specify this:
oapi_route_output(plug, 1..15, 0);.  Or perhaps it is just policy that if a
plug has multiple outs but only the first out is connected, all outs
re-route to out0.  Needs more thought.

> > 	int oapi_channels_min;
> > 	int oapi_channels_max;
> 
> I'm pretty confident this is not the right way to go. If it matters to you
> what output belongs to what channel then well known labels would be
> better, I think.

I wanted to make sure that this API handles mono plugs, stereo plugs, 5.1
plugs, and whatever turns up next without re-write.  I kind of assumed that
1 channel = MONO, 2 = LEFT,RIGHT, 6 = FL,FC,FR,RL,RR,LFE or some such
mapping.  Again the idea being that a plugin may handle more than 1 channel
in different ways, and that each in/out may not have the same # of channels.
A multi-pad drum machine may have a stereo snare, but a mono kick.  Maybe
this is not a good idea, I haven' decided.  That's why I sent it here for
discussion :)  Can you expound on the well-known labels idea?  I'm not sure
I fully understand what you're thinking.

> > 	 * create: instantiate the plugin
> 
> Maybe better to stiuck to LADSPA nomenclature? It might not be perfect,
> but it would avoid confusion.

Mmm, perhaps.  

> > 	/*
> > 	 * set_rate: set the sample rate
> 
> LADSPA definatly got this right, make it an argument to instantiate.
> Changing the sample rate of system is not an RT operation, so theres no
> need to make it ultra efficient, and instruments will have a very hard
> time rejigging everything. Simpler to just remove and instatiate a new one
> with the same settings.

ok, need to rework this then.  I had done it that way but decided this was
simpler, but I suppose you are correct :)

> > 	int (*oapi_voice_on)(void *plugin, int note);
> > 	int (*oapi_note_off)(void *plugin, int voice);
> 
> I prefer this one.

I do too, but I don't want to have 10 APIs that are purely for instruments
and not effects.  Add to those two the need for a voice_ison() api for proper
voice accounting of things like triggered samples, and eventually a
voice_param() to change dynamic parameters per-voice (something I have not
introduced in this API yet).  I'd like to be able to say 'increase the
volume of this existing voice'.  Fruityloops does it, and I've found it
to be quite conventient for things like strings and pads.   FL does
Velocity, Pan, Filter Cut, Filter Res, and Pitch Bend.  Not sure which of
those I want to support, but I like the idea.

> After reading this I think the best solution is something very like LADSPA
> but with note_on and note_off calls. Other useful things like enum types
> /can/ be added via external metadata, eg RDF ;)

Lots CAN be added via meta-data, but my goal was to be as simple as makes
sense, and no simpler.  Keeping LADSPA (imho) overly simple and using
external metadata is kind of ugly (again, imho).  We control LADSPA (well,
Richard does).  He can evolve it or change it and make a LADSPA v2 which
implements more advanced features.  BUT there are lots of reasons not too.
The idea of this API was to take advntage of all the great LADSPA plugins in
a more complex host.  One of the first plugins I will write to this API is a
LADSPA n-way adapter.  You can take a mono LADSPA effect and apply it to as
many channels as the host cares to pass it.  The host is simple.  The LADSPA
is simple.  Complexity hidden in a plugin.

> From: David Olofson <david at olofson.net>
> 
> Yes. Though, I think using a virtual instruments capable API for 
> effects can be great, especially when it comes to automation (both 
> need to deal with accurate timing and control interpolation), but of 
> course, there's always the ease-of-use/complexity trade-off; a 
> virtual instruments API will always be more complex than LADSPA.

I don't think it needs to be TOO much more complex.  The areas I've added
complexity:
* multi-in/out
* multi-channel
* type-specific controls
* voice_on/off

> Well, I'm not sure it's worthwhile going beyond floats + "hints"... 
> Strings could be interesting, as but how should the host deal with 
> them? Are they file names, just names, scripts, or what? How to edit 
> them? How to store them in "preset files" and the like?

String is a primitive - the control flags can specify a filename or a
directory name.  I'm sure that filename will be the most common case.  It
seems like String controls probably do not want to be automated.  But I see
no reason they CAN'T be.  All this complexity is to make things that plugins
want to say (enumerated values, integral values) less complex and less
hacked on the back of something else.  As simple as makes sense and no
simpler.

> I have a feeling that going down this route either results in complex 
> hosts, or a pointless "official" API for stuff that only matters to 
> plugins and their custom GUIs anyway - or both.

Custom GUIs don't even need to identify filenames as controls - they can do
it themselves.  The point of making it a control is to allow them to NOT
have a custom GUI

> How about just supporting a "raw data block" control type, so that 
> hosts can store the data in preset files or automation, but without 

I don't get what you're devolving it to.  The point of Strings was to say 'I
want a filename' or 'Any string will do for this field named SPEECH'.  Maybe
a raw data block is useful, but I haven't seen an example.  The types I
specified are based on experience using FruityLoops and similar apps and
dealing with VST instruments and other plugins and trying to proived the
simplest subset of what they need without oversimplifying.

> plugin binaries here. (I think we have concluded long ago that GUI 
> and DSP code should be entirely separated.)

absolutely.

> If it has outputs, it's a source. If it has inputs, it's a sink. If 
> it has both, it's (probably) an effect. Either way, the relation 
> between input and output can be of *any* kind (within the 
> restrictions of the API), so there's not much point in trying to 
> describe it to the host.

As above, I think I agree here.

> Likewise with timestamped "MIDI style" control events - either you 
> accept them, or you don't. Whether you have audio inputs, outputs or 
> both is another story entirely.

In my worldview MIDI is ENTIRELY the host's problem.  The
voice_on/voice_off/voice_change API is intended to provide the majority of
MIDI controlability.  The host can translate from actual MIDI.

> I'd definitely go for Unique IDs + external data here. Oh, and it's 

but why?

> probably a good idea to structure the ID as "vendor" and "product" 

I though about doing a Vendor Code.  But why burden some person with
distributing vendor IDs and maintaining that database, when storing a
string is just as simple.  It's not like a few bytes of memory are going to
matter.  Now, I'm NOT trying to be flippant about memory usage.  I just want
to be realistic.

> Well, I don't know Tim's reasons, I could give it a try: "Horizontal" 
> mixer modules, multipart synths and samplers and that kind of stuff. 
> Sure, you could probably just say you have 128 ins and 128 outs (or 
> whatever) - but then, would hosts assume that all are actually 
> initialized and in use? I see a risk of complicating port management 
> here, if ports can be "disabled" as well as "connected" or 
> "disconnected". It might be better if they just don't exist if you 
> don't need them.

Morte or less - yes.  A plugin sets it's maximum/minimum.  The host can use
or not use them.  A modular system (a la Reason) would let the user drag
cables.  A simple system would set up a lot of it automagically.

> Besides, I think fixed size arrays and the like are generally a bad 
> idea, and should be avoided whenever practically possible. You alway 
> hit the limit sooner or later...

Agreed 100000%  I refuse to fixed-size limit anything, if possible.  Hence
->oapi_ncontrols and ->oapi_controls

> > I'm pretty confident this is not the right way to go. If it matters
> > to you what output belongs to what channel then well known labels
> > would be better, I think.
> 
> Yeah, just as with input/output relations, there's no useful and 
> reasonably simple way of describing channel/input/output relations. 
> This just adds a dimension of indexing, for no gain.

Perhaps.  Perhaps different channels-counts per in.out is a bad idea.  But I
see a definate need for different channels-counts on input vs. output.  What
of a mono-izer.  Or a mono->stereo.  Or Stereo->5.1.

> Besides, you would at least have to have one pair of limits for each 
> input and each output for this to be of any use. (Unless all inputs 
> are required to be identical, and likewise for outputs.)

Limits are limits.  Current value is different.  The host already knows the
current value for everything but instruments.  It did the connecting.  I
admit to not having a good answer yet for asking an instrument how many
channels on a given out.  It could be simple: oapi_nchannels(plug, outnum);

> Consider a mixer, where you have insert sends and returns, as well as 
> bus sends and returns. Obviously, you'd probably want to be able to 
> use some 128 channels or so - but most probably not 128 busses!

I want to avoid pure hardware paradigms.  But you are correct in that it can
get complex.  Maybe it needs to be curtailed, but it definitely needs to
support multi-channel audio.  Ideas welcome.

> Here's another idea (from Audiality):
> 
> typedef enum
> {
> 	FX_STATE_CLOSED = 0,
> 	FX_STATE_OPEN,
> 	FX_STATE_READY,
> 	FX_STATE_PAUSED,
> 	FX_STATE_RUNNING,
> 	FX_STATE_SILENT,
> 	FX_STATE_RESTING
> } a_fxstates_t;

Similar to gstreamer.  Need to comtemplate this.  Can you elucidate on the
different states?  It seems like the same as multiple callbacks, just in one
callback.  How about params?

> the wrong order - and it's also strictly specified what plugins may 
> and may not do in each state, and when switching between any two 
> adjacent states.

can you post docs on this?

> Here, I prefer timestamped events, for various reasons. Most 
> importantly, timestamped events allow sample accurate control without 
> the host splitting buffers, and it allows plugins to implement event 
> handling in any way they like - including just executing all events 
> before processing each audio buffer, for quick hacks. (You could even 
> throw in a macro for that, for quick LADSPA-><new plugin API> 
> porting.)
> 
> I've been using this for a while in Audiality (very similar to the 
> event system designed for MAIA), and there's no turning back; 
> timestamped events are simply superior, IMNSHO.
> 
> Might look a bit more complicated than direct function calls at 
> first, but in my experience, it actually *simplifies* all but the 
> very basic stuff. Filtering, proccesing and routing of events can be 
> done by inserting plugins in between other plugins, and lots of it 
> becomes trivial, as you can disregard the timestamps altogether in 
> many cases.

Can you talk more about it - I don't know the paradigm.  I'm open to ideas
:)


I REALLY appreciate all the feedback.  I will incorporate the bits I
definitely agree with and hopefully I can get more discussion about the bits
I am unsure about.  I'm willing to be convinced if you are :)


Tim



More information about the Linux-audio-dev mailing list