[linux-audio-dev] PTAF link and comments

[David Olofson]

On Thursday 06 February 2003 12.58, Laurent de Soras [Ohm Force] 
wrote:
> Late reply, I was quite busy and english writing takes me
> time. I commented various people quotes here. However I
> haven't read recent posts yet, I'm a bit overhelmed by
> the reply avalanche. :)

Well, we're crazy about plugin APIs around here! ;-)


[...]
> > I'm questioning whether
> > having a simpler query based system may be easier.  I don't like
> > the idea that you have to instantiate to query, though.
>
> Many plug-in standards require instanciation before fetching
> properties. PTAF must be able to make good wrappers for existing
> plug-ins, industry requires it.

That's one point, although I think you could have the plugin factory 
of a wrapper do the instantiation "secretly". A wrapper *is* a plugin 
host anyway, so this shouldn't be a major issue.

Either way, for various other reasons, I'm beginning to think that 
there are other reasons to use lightweight plugin instances for 
metadata requests. We have already mentioned plugins that can 
"mutate", and generic wrappers (ie wrappers that have a control for 
selecting the alien plugin to wrap) belong in that category.


> > Yeah, I've thought about that too. Rather hairy stuff. The only
> > easy way to avoid it is to strictly define what the host should
> > tell the plugin before asking anything, and then just leave it at
> > that.
>
> We can also add somewhere a watchdog limiting the recursion.

Yeah, but it doesn't really solve the problem, does it? If you get 
endless recusion (or rather ping-pong, if we're talking about events; 
not much better though), it's because there's something wrong with 
the host, the plugin or both.

There has to be a way to guarantee that dependencies don't become 
circular... Can we have some actual examples to break down? I've 
never written a host that changes it's parameters based on plugin 
properties, so I've never seen the problem in real life.


> > [...the "sequencer" thread...]
>
> As suggested we can call it "host", or "main" thread. In my opinion
> generic forms of hosts have several "virtual" threads :
>
> - Audio: handles audio processing and related sequencing
> - Main/Background: handles generic operations, state transitions,

What state transitions?


> as well as background disk streaming, data preparing for audio
> thread, etc.
> - GUI thread (at least on Win and Mac OSs), can theoretically
> be merged with the main thread, but it's not recommended (Mac
> programmers have the bad habit to poll the mouse within a
> blocking loop when user is holds a click).

Well, they didn't have much choise in the old days... Win16 had the 
same problem. Provided we're talking about Mac OS X, it certainly is 
nothing but a bad habit, though.

Anyway, I know what you mean by these threads - but I still don't see 
it as relevant to a plugin API. A GUI-less audio server may well have 
only one thread, and either way, plugins should not be required to be 
thread safe, unless they explicitly ask for it (trouble, that is ;-) 
themselves.


[...]
> > Well, you might still be able to comment on an idea: Splitting up
> > the GUI in two parts; one that runs in the same process as the
> > DSP plugin, and one that runs in another process, possibly on
> > another machine.
>
> Yeah, I had this in mind when i was talking about lauching the
> GUI from the plug-in.

Yeah... Though I think all this should be left to the host, 
regardless. Just tell the host where your GUI is. If it's a "local 
half" plugin (that in turn refers to an external GUI), it could be in 
the same binary, as it's still just a plugin with a minimum of 
external dependencies. (Whether or not it wants to run in the audio 
thread is another matter. It can still use the same API - and the 
same rules WRT GUI toolkits still apply; don't use them.)


> > You would need a feature that allows control outputs to be marked
> > as "active" or "passive". This allows hosts to control how things
> > are handled.
> >
> > If I see a knob that is turning via some old automation, I
> > should be able to grab it in the UI and take over.  It is not
> > clear to me, however, whether the host should remember that and
> > leave control with the highest prio controller, or just make it
> > temporary.  This needs more thought.  It may not be the domain of
> > the API at all, but the host.
>
> But the token system is just about the same concept, isn't it ?

Not really. The difference is that my approach doesn't require GUIs 
and hosts to use a special API just for talking to controls.

Also, tokens don't mix with the direct connection system we're using. 
(XAP plugins generally don't send events through the host, but rather 
directly to the input queues of other plugins.)


> For a given parameter, there is at most one active client at a
> time, others are passive and receive only change notifications.
> Arbitration is done by host, who is allowed to deprive a client
> from his token to give it to a client requesting it.

This doesn't work if some clients are out-of process. Control outputs 
with active/passive modes do, since you never have to inform client 
whether or not their data is used.


> For example it makes sense to give less priority to the
> automation system than to user via GUI, so the latter can
> steal its token, just by requesting it. When user has released
> the knob, it give back the token to the host making it
> available for any use (can transmits it implicitly to the
> automation system).

Sure, but it just seems like a more complicated way of doing it, 
especially when threading and out-of-process issues are taken in 
account.


> > I actually think having to try to *call* a function to find out
> > whether it's supported or not is rather nasty...
>
> Good point.
>
> Anyway I would like to minimize the number of functions.

Good idea, of course. And besides, this even further reduces the need 
for "advanced" methods of maintaining supported and unsupported 
calls.


> Functions are handy when the calling conventions
> of both parts are compatible. If the language you use
> doesn't support this convention natively, all functions
> have to be wrapped, which is a real pain.

Really?

Either way, we also have to consider the importance of wrapping the 
API. How many write VST hosts and plugins in other languages than 
C++? (I know that some do. FruityLoops is written in Delphi, IIRC.)

We also have to consider the balance between the work required to 
implement a wrapper/language binding and the impact simplifying that 
has on the native API, in terms of performance and ease of use.


> More, there are still issues with existing plug-ins, reporting
> wrongly if they support some opcode or not (talk to this issue
> to Plogue people :).

Bugs are always bugs. A simple testing host in the SDK would eliminate 
all valid excuses for this particular bug. (And it could of course be 
included in the host SDK as well, to warn every user about broken 
plugins.)

Either way, I don't see a real problem here. Just don't call a NULL 
pointer, ok? :-)


> > I think this separation is a mistake. Of course, we can't have an
> > API for complex monolith synths scale perfectly to modular synth
> > units, but I don't see why one should explicitly prevent some
> > scaling into that range. Not a major design goal, though; just
> > something to keep in mind.
>
> The separation between coarse/fine grained plug-ins was just
> a design goal for the API. Fine grained plug-ins would have
> more issues, like the ability to do fast sample-per-sample
> processing (for feedback loops).

Yeah, but you can't really get around that. You can still implement a 
host that does small block (not the Chevy V8! :-) sub nets for 
feedback loops, but it's going to be rather expensive. OTOH, that's 
*always* expensive...

My point though; expensive is something you could live with. It might 
even matter to you. *Impossible* is a different story entirely.


[...]
> > How about defining buffer alignment as "what works good for
> > whatever extensions that are available on this hardware"...?
>
> Good, but it requires the host to be updated when new hardware
> is released.

Or just make it a setting in "advanced options"...?

(If you're not very interested in users upgrading your software, that 
is! ;-)


> We can add a host property indicating the current
> alignment so the plug-in can check it before continuing.

Very good idea. There *are* platforms where a plugin could crash the 
host if they ignore this.


> > you'll get a lot of arguments about that on this list - linux
> > people tend to have a 486 or pentium stashed somewhere. :)
>
> Professional musicians and studios have recent hardware, at
> least it's what survey results show us.

They don't really have much of a choice, do they?


> Recent audio software
> produced by the industry also tend to be hungry, requiring
> fast hardware - I don't say it's good, it's just a fact, and
> industry is the main target for PTAF.

It's a fact, and it's not as much a result of inefficient APIs as it 
is a result of people wanting "cooler" DSP effects. I don't have a 
problem with this, but I *do* have, and always will have, a problem 
with wasting resources for no good reason. Thus, I tend to look for 
optimal solutions at all times. If it's something I'll have to live 
with for years, I look harder - just in case it turns out that it 
actually matters.


> If you plan to make a standard, it's better to build it for
> having use over years. After all, MIDI is still here and will
> probably last 20 years again. OK things are a bit different
> with pure software protocols, but arguing little performance
> boost against obvious programming safety is pointless for me
> (sadly programmer's brain doesn't follows the Moore law).

Very good points, and of course, I'm not proposing that we should make 
life hard for plugin authors just to save a few cycles.

It's just that the wasted cycles is not the *only* problem I have with 
the dispatcher approach, so if there's a better solution (that also 
happens to be a bit faster), why not use that?


> > Agreed - but they also attach value to efficient software.
>
> Yes, but given my experience with customer support, unsafty
> displeasure is one or two orders of magnitude above efficency
> pleasure.

Very good point! If it *crashes*, who cares if it's slightly faster?


> > That's one way... I thought a lot about that for MAIA, but I'm
> > not sure I like it. Using events becomes somewhat pointless when
> > you change the rules and use another callback. Function calls are
> > cleaner and easier for everyone for this kind of stuff.
>
> The advantage of events is that you can have several of them at
> once, giving the plug-in the possibility to optimise operations.

Yes.

As a matter of fact, I have *exactly* this problem with the FX plugins 
in Audiality. Initializing the parameters of the xdelay or reverb 
effects means they have to recalculate all taps almost once per 
parameter. *heh* They're about the last controls to use function 
calls, and it's a real problem.

I could handle it by implementing the changes in the process() call, 
though, just like with events. It's just that the changes are not RT 
safe, so they have to send the work off to another thread. Dealing 
with that *and* the function call API, well... I'll just leave it 
until I've fixed the plugin API.


> > We've just spoken of calling the process() function with a
> > duration of 0 samples. Events are then processed immediately,
> > and no second API is needed.
>
> Dunno if this factorisation is really good. Different goals,
> different states for call, it should be a different opcode/func.

There's a *big* hidden issue with thinking about different interfaces 
yor different states. States are related to what context a plugin 
runs in, and using different APIs in different states doesn't really 
help. You can't perform non RT operations on a running plugin just 
because it has a separate interface for it! Either the plugin must be 
thread safe, or the host has to take the plugin out of the RT thread 
first.


> > Anyway, I'm nervous about the (supposedly real time) connection
> > event, as it's not obvious that any plugin can easilly handly
> > connections in a real time safe manner.
>
> Don't mix up audio configuration (which is not real-time)
> and connections. In most of cases I see, (de)connections
> is reflected in the plug-in by changing a flag or a variable,
> making the processing a bit different. For example a plug-in
> has a 2-in/2-out configuration. When only one input or output
> channel is connected, it gives the ability to make the
> processing mono, or to (de)activate vu-meters, etc.

You're totally right. I wasn't thinking straight.


[...]
> > This is where it gets hairy. We prefer to think of scale tuning
> > as something that is generally done by specialized plugins, or by
> > whatever sends the events (host/sequencer) *before* the synths.
> > That way, every synth doesn't have to implement scale support to
> > be usable.
>
> That's why Base Pitch is float. But because MIDI is still there,
> existing plug-ins will be happy to round (after scaling if 1/oct)
> Base Pitch to get the note number.

Yeah, I realize that, but there's still a difference between explicit 
1.0/note and 1.0/octave with the scale assumed to be 12tET. If the 
scale *isn't* 12tET, rounding Base Pitch won't work as expected.


> > As it is with VST, most plugins are essentially useless to people
> > that don't use 12tET, unless multiple channels + pitch bend is a
> > sufficient work-around.
>
> VST has also a finetuning delivered with each Note On.

Yes, I remember now that you say it. However, it has very limited 
range (cents; sint8_t), and the MIDI pitch field still doubles as the 
"note ID", so this is in no way a substitute for continous pitch, or 
even sufficient for handling scales that are substantially different 
from 12tET.


[...]
> > The only part I don't like is assuming that Base Pitch isn't
> > linear pitch, but effectively note pith, mapped in unknown ways.
>
> However many (hard/soft) synth can be tuned this way. It is a
> convenient way to play exotic scales using traditional keyboard,
> piano roll or other editors / performance devices.

What's wrong with using a generic scale converter plugin *before* any 
synth you want to do this with? That's one of the major benefits of 
using continous, linear pitch, IMO. I don't think synths should have 
internal scale converters, unless they have *very* good reasons. (And 
I can't think of any.)


> > Right, but MIDI is integers, and the range defines the
> > resolution. With floats, why have 2.0 if you can have 1.0...?
>
> To make 1.0 the middle, default position.

Yeah, but what is a "default" velocity in real life...?

It's something that was invented for the MIDI protocol, to support 
controllers without velocity sensitivity. As even the cheapest "toy" 
MIDI controllers are velocity sensitive these days, it is essentially 
irrelevant, even to MIDI devices.


[...]
> > Anyway, what I'm suggesting is basically that this event should
> > use the same format as the running musical time "counter" that
> > any tempo or beat sync plugin would maintain. It seems easier to
> > just count ticks, and then convert to beats, bars or whatever
> > when you need those units.
>
> But how do you know on which bar the plug-in is ?

Somewhat like in VST; I ask for a time info struct.


> Tempo and
> time signature can change in the course of the song and it
> is not obvious to deduce the song position from just an
> absolute time reference or a number of ticks or beats.

That's why tempo and beat sync plugins in VST have to ask for time 
info for every block.

The only difference with XAP is that if you have input controls for 
musical time, you'll get timestamped events whenever interesting 
changes occur, so if you just listen, you can have a sample accurate 
view of the musical timeline at all times.

(XAP beats VST in this regard, since VST can only give you time info 
for the first frame in each block. XAP plugins can maintain sample 
accurate beat sync even if there are several *meter* changes within a 
single block. All VST can tell you is the tick time for points in 
audio time, so you could miss meter changes.)


> > Yes. And bars and beats is just *one* valid unit for musical
> > time. There's also SMPTE, hours/minutes/seconds, HDR audio time
> > and other stuff. Why bars and beats of all these, rather than
> > just some linear, single value representation?
> > Could be seconds or ticks, but the latter make more sense as it's
> > locked to the song even if the tempo is changed.
>
> For reasons mentioned above, both time in second an musical
> representation are needed.

I'm not denying that.


> SMPTE etc can be deduce from the
> absolute time in seconds or samples.

Yep, but that's true for musical time as well - although musical time 
*is* indeed of much more interest to many more plugins, so 
special-casing it in the API is probably motivated. (Which is why XAP 
has a set of controls specifically for musical time.)


> > Accuracy is *not* infinite. In fact, you'll have fractions with
> > more decimals that float or double can handle even in lots of
> > hard quantized music, and the tick subdivisions used by most
> > sequencers won't produce "clean" float values for much more than
> > a few values.
>
> But is that so important ? Float or int, we manipulate numbers
> with a finite precision.

Sure, but multiplication of two integers always gives you an exact 
result. (Yes, even if it doesn't fit! It's just that high mid and 
level languages can't make use of the upper half of the result - 
which I find really rather annoying... *heh*)


> And if someone is happy with tick
> scales multiple of 3, someone else will complain about not
> supporting multiples of 19, etc.

Yeah, that's why sequencers use these "weird" numbers... They don't 
cover *everything* of course, but they're high enough that rounding 
times to the nearest tick is ok - and then you can keep calculations 
in the sequencer exact.


> For the drifting question, it is neglictible with double, and
> can be easily resync'd if required. Ohm Force's plug-ins use
> double as positions and steps for the LFO phases and it works
> fine without resync during hours, on any fraction of beat.
>
> If you need exact subdivisions, you can still convert the
> floating point value into the nearest integer tick.

Well, I can't disagree, as I used to argue for your position, and have 
yet to see actual proof of the rounding errors/drift being a problem. 
Accurate musical time calculations belong in the sequencer domain, as 
we can't support them anyway, unless possibly if we switch to musical 
time timestamps.


> > The host would of course know the range of each control, so the
> > only *real* issue is that natural control values mean more work
> > for hosts.
>
> No it just would store and manipulate normalized values.
> Natural values would be needed only for parameter connection
> between plug-ins, if user chooses to connect natural values.

There will be conversion work to do *somewhere* no matter what. 
However, I don't like the idea of making all controls have hard 
ranges. It restricts the users in ways not necessarily intended by 
plugin authors, since there's no way to set controls to values 
outside the range the plugin author decided to use, even if there's 
no technical reason why it couldn't be done. You're basically 
throwing away interesting possibilities.


> > What's wrong with text and/or raw data controls?
>
> They often require a variable amount of memory, making
> them difficult to manage in the real-time thread.

Yes, I'm perfectly aware of that... *heh*


> However there is a need for this kind of stuff for sure.
> I thought more about encapsulating them into specific
> events.

Yeah, that's the approach I have in mind. Chained fixed size blocks or 
something... It's not too much fun working with, but we *are* dealing 
with real time here; not word processing. People wanting to write 
real time software will simply have to accept that it's a different 
world, since there's no way we can changes the laws of nature.


> > 1) none - host can autogenerate from hints
> > 2) layout - plugin provides XML or something suggesting it's UI,
> > host draws it
> > 3) graphics+layout - plugin provides XML or something as well as
> > graphics - host is responsible for animating according to plugin
> > spec 4) total - plugin provides binary UI code (possibly bytecode
> > or lib calls)
>
> I agree with 1) and 4). For my experience, when a plug-in
> has 4), it has also 1), but it's not *always* the case
> because developers don't feel necessary to implement the
> hints correctly (dev time matters). Adding more UI
> description is likely not to be supported by either host
> or plug-in, making them useless.

Some of it can be made optional, of course. In fact, even range and 
default values could be made optional. Just define the default range 
for an unmarked control as [0, 1], and the default value as 0. 

Whether the range should be soft or hard, I'm not sure, but soft seems 
more useful.


> Good solution would
> be using portable UI libs, as someone suggested here,
> but everyone knows it's the Holy Grail.

Yeah... *heh* Even the nicest toolkit/scripting solution will only 
work for *most* plugins; not the ones that want new, cool visual 
effects.

Though SDL is as portable as it gets, and what you can't do with that 
can't be done, basically. I can assure you that an API like that 
*will* work for any plugin GUI, one way or another.

How? All it provides is basic blitting and pixel level access to the 
screen and other surfaces. This is stuff that any platform with 
graphics can support, and most GUI toolkits also have some form of 
low level canvas or other component that can do it. I've even 
implemented the SDL 2D API on top of OpenGL (glSDL), for lightning 
fast rendering where h/w acceleration is available. (Pretty much the 
only way to have fully accelerated SDL on X at this point.)

If we want to spend time on anything along the lines of a portable GUI 
toolkit, SDL's graphics functionality is the *first* thing we should 
implement, or it's just a waste of time. Then we can start thinking 
about building something like VSTGUI and/or other stuff on top of 
that.


> > The spec should not dictate ABI, either.  The ABI is an articat
> > of the platform.  If my platform uses 32 registers to pass C
> > function arguments, it is already binary imcompatible with your
> > PC :)
>
> The only way to guarantee real portability through platforms
> and programming languages is to describe the API at the byte
> level. This include calling convention. IMHO We can assume
> that every target system has a stack, and can pass parameters
> on it.

Good point. I think we can count on standard C calling conventions to 
work and be supported by all languages on every platform. No big deal 
if it isn't as fast as the default conventions - the heavy traffic 
goes through the event system anyway, and there are no per-event 
function calls.

Of course, a language that cannot interface with the event struct 
would be a problem. Requirements: Pointers, 32 bit integers, 32 bit 
floats and 64 bit floats. (We *could* make all integers 64 bit on 64 
bit CPUs, but it seems insane and pointless to me. 64 bit machines 
don't have infinite memory bandwidth...)


[...]
> If you want to generate only ASCII in your program, just write
> only ASCII, it will be supported by UTF-8. However if you have
> to read strings, it's better to be aware that it's UTF-8,
> because incoming strings may contain multi-byte characters.

...and what that means to your average plugin is basically "Don't 
start splitting strings arbitrarily!"

Would be interesting to know which ASCII values are valid inside 
multibyte charatcers, BTW. Is there a risk you'll see false slashes, 
colons and things like that in paths, if you don't parse the UTF-8 
properly? (There isn't IIRC, but I'll have to read up on this.)

[...]


//David Olofson - Programmer, Composer, Open Source Advocate

.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`---------------------------> http://olofson.net/audiality -'
   --- http://olofson.net --- http://www.reologica.se ---