Re: [linux-audio-dev] Plugin APIs (again)

On Saturday 07 December 2002 20.12, Tim Hockin wrote: I think we're actually talking about *three* kinds of controls, while everyone is still talking about their own set of two kinds of controls. What I'm thinking is: Master Controls: Whether you have multipart/multichannel instruments/plugins or not, these always address the instrument/plugin instance as a whole. Ex: Master output volume of a multipart sampler. MIDI: System commands, some SysEx. Channel/Part Controls: These address a specific Channel/Part of an instrument/plugin. If the instrument/ plugin has only one Channel/Part, these messages can be considered equivalent to Master Controls. Ex: Channel dry send volume. MIDI: All events with a channel field. Voice/Note Controls These address specific Notes or Voices, to control them individually while they are playing. Whether or not a Note or Voice is a single oscillator or a network of objects is irrelevant to the API - this is an interface to an abstract object. Ex: Note Aftertouch. MIDI: NoteOn, NoteOff, Poly Pressure, various Universal SysEx messages. I don't think this has much to do with the API - and I don't see the possibility of a clash. If you take any two controls, they may have any relation whatsoever, *defined by the synth*, and the host should assume nothing about that relation. Whether the two controls are both VOICE, both MASTER or one of each, should not matter - the synth still decide what their physical relation is. So, in your example, you simply have two controls; MASTER::VOLUME and VOICE::VOLUME. The synth may multiply them internally, or apply MASTER::VOLUME in the output mixing loop, or whatever. Two controls, two values - the two controls just happen to have the same hint: VOLUME. You're thinking entirely in keyboardist terms here. :-) You're forgetting that some instruments really don't have anything like attack and release in real life, but rather "bow speed", "air speed" and that kind of stuff. I think I've suggested before that you may replace the velocity "control" with a continous control like "bow speed" (what was that generic name I came up with?) - which, if you think in keybordist terms would correspond to key "position". Derive that at the point of contact, and you have your velocity. You could then indeed think of velocity as a continous control - it's just that your average keyboard controller only changes it when you press or release a key. So, given that controllers and instruments are different, it might be a good idea to support *both* "velocity" and "position" (pick one for each patch/sound!) - and both would be continous, of course. Well, I agree with you - velocity is *not* loudness. (There are other controls for that kind of stuff; just look through the "standard" MIDI CC list.) Well, you won't get *me* to accept it that easy. ;-) [...] I think so, but I'm not 100% determined. (Audiality is still mostly a monolith, and I'm not completely sure how to best split it up into real plugins.) Yeah, but what if you want non-trivial interaction between the parts/channels? (Drum kits or something...) And either way, you may be able to optimize things better if you're able to run the loops over parts/channels where appropriate, which is something you cannot do if the parts/channels have to be separate instances. If that is prevented, I think VSTi and DXi will kill us before we even get off the ground. Many users *like* plugins that act and look like real hardware devices, and I don't think there's much point in trying to reeducate them. Remember that lots of people actually come from the hardware world. This is not to say we should hold back inovation and proper software design in favor of emulating hardware. I just believe that preventing plugins from being multipart doesn't make much sense. Just reserve one control/event port for the whole instrument/plugin, and then allow the instrument/plugin to have as many other control/event ports as it likes - just like audio in and out ports. Well, there actually *are* things very similar to "instrument plugins" in Audiality (although they don't process audio; only events): Patch Plugins. Each Patch Plugin instance has an input event port. One argument of all events sent to Patch Plugins (or rather, to Channels, which is where Patch Plugins are plugged in) is used as a "tag" field, which polyphonic Patch Plugins let you use to reference individual voices. The MIDI->Audiality mapper just throws the MIDI pitch into this field, and -1 ("all") when there is no pitch. As to master events, there are none of those at this point, since the FX insert plugins - for some reason - don't use the event system for control yet. The organ doesn't have velocity at all - but the violin, and all wind instruments I can think of, would be examples of instruments that indeed *have* "some concept of velocity" - but one that is very far from on/off with a velocity argument. Any plugin that processes events rather than (or in addition to) audio. They're usually called "MIDI plugins". In some cases, that might make sense, yes. Where? They have to get the VVIDs from somewhere anyway. It's probably not a great idea to just assume that plugins can accept any integer number as a valid VVID, since that complicates/slows down voice lookup. In fact, Audiality *does* take whatever "voice tags" you give it - but as a result, it has to search all voices owned by the channel to find the voices. *heh* Could be optimized, but why not just use an array of "Virtual Voice Info" structs; one for each valid VVID? Anyway, dividing a range of integer numbers seems pretty trivial to me - but it should still go into the host SDK lib, of course. :-) So you want to start sorting *everything*, just so people can write *slightly* simpler event sending code in some special cases? I'd much rather require that events sent to a port are sent in timestamp order, so the host can just merge sorted event lists. In cases where you have multiple connections to the same event port, you just use "shadow ports", so the host gets a number of ordered event lists that it merges into one just before running the plugin that owns the real input port. Problem is that if that ID is the actual number of a real voice, you can't steal a voice in the middle of a buffer. You have to wait until the next buffer (or longer, if you're running in different theads, or over wire), so the event sender has a chance hearing that "hey, that voice now belongs to some other note!" If the ID is decopled from physical voices, you (as a sender) can just go on with your business, and you don't *have* to know whether or not your IDs are still valid before sending events. The synth will keep track of the *whole* voice allocation matter; not just part of it. Yes, but see above... This problem is real, and I still have it in the lowest levels of Audiality, where it - fortunately - doesn't seem to matter all that much. (Internal buffer size is restricted, so it voice allocation granularity isn't all that bad.) But consider what happens if you have voice related IDs on the instrument API level, and try to play a fast series of short notes. Since the plugin can't know for sure how long these notes will need their voices, it simply has to give you a now voice for each note. While, if you use VVIDs, you'd just pick a new VVID for each note, and let the synth manage the allocation of actual voices. If the notes are short enough, voices can be reused within a single buffer or roundtrip cycle. No. It's a matter of whether or not to depend on roundtrips and buffer splitting to avoid waste of resources. Why complicate things? With a protocol that doesn't entirely depend on short round-trip latency for proper operation, you can just tunnel the events directly to the receiver, possibly translating the timestamps if you're not in sample sync. (Which you should be if you're serious. Get proper hardware. :-) Allocation is just a matter of preparing the plugin for receiving and handling a specific number of VVIDs. You could just say that there are always 65536 VVIDs, but that sounds both wasteful and restrictive at the same time to me. As to the implementation; if your plugin does what Audiality does right now, the plugin may ignore the VVID allocation requests entirely and just say "ok, whatever" - since anything that fits in the VVID field is ok. (Audiality just throws the VVID into the voice struct of the voice it actually allocates, and then looks for it when it receives events talking about it.) Definitely. In fact, applications that don't support plugins that *send* events are frequently referred to as broken these days. Well, if your threads are not in sample sync, you're in trouble... But indeed, the host can deal with that as well, if for example, you're crazy enough to want to use dual, unsynchronized or "incompatible" sample rates. :-) Timestamps matter a lot if you're going to do anything much at all with the events. (Record them or convert them to MIDI, for example.) That said, replies to requests (such as VVID allocation) should "bypass" the normal events. (Not that it would make much of a difference if you process them right after the plugin has returned from process() or at the end of the engine cycle. It may matter if you're piping events back and forth over wire, though.) It doesn't *have* to be - but if you look at the MAIA or Audiality event systems, you'll realize why I strongly prefer fixed size events. It's all about performance and complexity. Note that you still *can* pass around data blocks that won't fit in an event. Just pass data blocks by reference. (I've designed protocols for that for MAIA, if you're interested in the details.) What do you need them for? I have multiple parameters in Audiality events, but it seems like I'll need *less* parameters once I've cleaned out some silly MIDIisms. (Such as having pitch and velocity in "start" events.) Exactly. Especially considering that you'll send *multiple* events for every "voice_start", and that there may be a lot of control changes while playing real music, performance matters. We can probably forget about anything like traditional dynamic memory allocation. You *could* a few different event sizes - but then you'd have to specify size when allocating them, and the deallocation code would have to check some hidden field or something, so see how big every event is, in order to put it back in the right pool. (This was actually written by Steve Harris, AFAIK. :-) Yes. Let's just don't forget that although we may need to design something new from scratch to get a nice and clean API, we do *not* automatically have to throw away the feature set of LADSPA. We should basically have LADSPA functionality, but without LADSPA style control ports, and with event ports + "abstract controls" instead. Some of the "negotiation" calls may be replaced by events. (MAIA philosophy again.) //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---