On Thursday 12 December 2002 13.32, Steve Harris wrote: Right. So, what we need to do now is agree on a sensible time struct. That is, basically copying that of VST or JACK. I'm trying to find out whether or not it makes sense to say anything more than "in the past" or "in the future" about timestamps that are outside the buffer... Do you ever have a *valid* reason to query the musical time of an audio time that is outside the time frame you're supposed to work within? For example, if you're delaying events by musical time, you're not *allowed* to keep the audio time of a future musical times, since it may be invalid by the time you get there. (Timeline edits or transport manipulation.) So, you only really have to know that the event *is* in the future, so you can put it in your internal queue for processing later on. Anyway, the *real* reason why I'm worrying about this is that such a rule could make life a lot easier for hosts. You can cache time structs for the current buffer, but never have to even generate them for timestamps that fall outside. (The call would just return one of two error codes or something.) //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Thursday 12 December 2002 15.32, Steve Harris wrote: Yes... Maybe one should actually have specific functions that work with *durations* instead? Or maybe not - they would still be completely equivalent to asking for "now" and "now + delay", and then calculating the delta. It would just be a minor performance hack that the host may or may not be able use for some optimizations. Either way, do keep in mind that whether you ask for a duration or two points on the timeline, the result is still invalidated if the timeline is changed or there is a transport skip during the period of time you're dealing with. There's no way to avoid that without using musical time (or other timeline relative) timestamps - and then you're in that nice position where you can lose events forever due to timeline loops and transport skips. *heh* Thinking about situations when this delay time thing would matter, tempo scaled envelopes spring to mind. Say you want something to decay linearly in exactly 1 bar, tracking tempo and tempo changes. You *could* just ask for the audio time for "now + 1 bar" and the go ahead. Simple - calculate the slope and then go back to processing. But then you would not even track tempo changes properly, let alone transport skips! So, it doesn't work anyway. The *correct* way would be to look at the musical time for *every single sample frame*, calculating the level directly as a relation between the starting time and the current time. (And then you would obviously not need to see into the future at all; just find out the musical time of the current sample frame.) Any other approach is an approximation. (Note that you may still implement it without one host call per sample frame! Just track tempo changes, and assume that the slope only changes when they occur. For beat synchronized effects, you'll also have to look for transport skips, since tempo changes do not reflect those.) Now, a relatiely nice approximation (for reasonably small blocks and not too drastical tempo changes) would be to check start and end times, but restrict the latter to within the current buffer. You still won't handle tempo changes or transport skips in the middle of the buffer properly, but at least, nothing unexpected can happen in between your start and end points. So, where's the big difference? Well, no one can edit the timeline while you're in the middle of processing a block. As soon as the host starts running all plugins for one block, all events must be set in stone, or you will have parts of the net running out of sync! This applies to transport as well. A skip would already be known and "implemented" before your plugin is called. No surprises, that is. OTOH, *no one* knows what will happen in the next block! So, whatever you ask the host about regarding the time after the current block, it may well be totally nonsense when you're actually there. I'm becoming more and more convinced that there is no valid reason for a plugin to ask detailed questions about the future. The answers are just (qualified, but still) guesses anyway. No, I'm just trying to stop plugins from asking about the future, and then assuming that what they asked is still true when they get there. If by extrapolation, you mean figuring out things like "when is 1 bar ahead?", you're on slippery ground... You can still do that, but in effect, you'll just be *guessing* - just like the host would, if it would answer questions about the time beyond the end of the current block. I'm trying to a avoid an API where you have to think twice to know if something the host tells you is a *fact* or an *estimate*. //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Fri, 13 Dec 2002, Tim Hockin wrote: It is tempting but I object to having high-level concepts such as musical time in XAP. It should be the host's responsibility to set and update the proper delay time for a tempo delay effect. We need another, completely different plugin system for sequencing and composition. -- Sami Perttu "Flower chase the sunshine" Sami.Perttu(a)hiit.fi http://www.cs.helsinki.fi/u/perttu

ardour uses 1920 ticks per beat. if the beat is 180 bpm, that's 5760 ticks per second, or 1 tick every 8 samples at 48kHz. thats a ridiculously dense event stream when most of those ticks have no significance at all. if you make ticks_per_second constant, then ticks_per_beat is not constant. if ticks_per_beat is constant, then ticks_per_second is not constant. computing the number of ticks between two arbitrary times is (theoretically, at least) quite complex. imagine a situation where a piece shifts time signatures a few times, has some decelerando and accelerando sections and a few sudden tempo changes (think prog-rock, for a start :). how many ticks between a given point in one "section" and "another" ? if you make ticks_per_beat constant, its easy to compute the tick count, but hard to compute the interval between ticks. if you make ticks_per_second constant, its easy to compute (hah!) the interval between ticks, but knowing how many ticks are between two musical times is fairly tricky. i like this basic idea, but i don't think that you can send one event per tick. i think that david is not far from the mark. i think it should probably work more like this: Plugin: i have an event at "bar|beat|ticks", what transport frame is that ... [ computes ]. Host: sends a running audio frame count Plugin: defines a particular audio frame count as "zero" on the transport frame scale (e.g. when playback starts), then executes the event at the correct offset. when the plugin considers the transport to be not moving, no events get executed. something like that. the hard part here is that plugins needs to be able to share the tempo map, because they need to be able to look ahead to compute an audio frame (offset) that corresponds to a given B|b|t time. this is going to be a critical problem with JACK as well. its all very well saying that a particular plugin can do this, but if you want several plugins operating in sync with the same notion of tempo, there needs to be a way to share the information between them. VST does this in a rather crude way, JACK currently doesn't do it at all. if you put it in the host, that helps a lot, but it now requires ways to edit the tempo map, which would typically be in the domain of a sequencer, and the vibe here is that the host is not necessarily a sequencer ... --p

On Friday 13 December 2002 10.11, Tim Hockin wrote: Yes, that's an easier and much more robust way of doing it. As soon as ask for absolute musical time, you ask for trouble - and you *have* to deal with it, or Bad Things will happen as soon as people use your plugins together with loops, conditional jumps in interactive scores and that kind of stuff. Maybe... It's not perfect, though, but if you count ticks *and* keep track of tempo change events, you *can* actually maintain your own, exact copy of the timeline, in whatever format you like. And, of course, this works even when you get input from different timelines on different channels. No, that can change at any time (or many times) in the block. Yes. Plugin SDK macros can be provided to implement this in one or two generally useful ways, if that makes sense. Yes, although sending one event for every tick, assuming that a tick is the smallest time unit that the sequencer knows about, means that you get lots of tick events, or that you need to do some extra work in plugins, to calculate the ticks that don't come in as events. Right; plugins that listen only to ticks when they actually come probably aren't interested in sub-tick accuracy anyway. Plugins that *are* will work fine regardless of tick resolution, since they need to generate a full sample frame rate model of the timeline internally anyway. This should work. Yes; a "mini sequencer toolkit", basically. Well, you seem to have solved the problem with multiple timelines in the same net. You can have one per Channel this way. :-) *hehe* //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Friday 13 December 2002 22.14, Tim Hockin wrote: Yes - as long as the song position doesn't skip, because that won't (*) result in tempo events. Plugins that *lock* (rather than just sync) must also be informed of any discontinuities in the timeline, such as skips or loops. (*) You *really* don't want two events with the same timestamp, where the first says "tempo=-Inf" and the other says "tempo=120 BPM". But that would be the closest to the correct way of describing a transport skip you can get. Next is "running like hell" for one sample frame, and then reverting to the right tempo, but that's a *really* nasty thing to do to plugins that are only concerned with tempo... That's a good way to explain what prequeueing is really about. :-) No. I thought some about this earlier, but forgot to write it. This is all you need to maintain a perfect (almost - read on) image of the timeline: Tempo changes Whenever the tempo changes, you get a sample accurate event with the new tempo. Unit: Ticks/sample Position changes Whenever the song position changes as a result of something other than the usual tempo pushing it forward, you get an event with the new absolute song position. Unit: Ticks Meter changes When PPQN (who would change *that* in the middle of a song...?), meter, etc changes, some plugins will want to know this, because it affects the way they interpret musical time. This event is probably best implemented as a notifier that gives you a pointer to a static struct owned by the sender of the event. (The sequencer, the timeline plugin or the host.) (Obviously, this means that the sender will have to maintain multiple such structs if there are mid-buffer changes or multiple changes per buffer!) Unit: *XAP_time_info That's it! That's all there is to know about the timeline. And you don't have to be flooded with any of these events (unless they user is abusing the timeline); you'll only get them when something interesting happens. Now, there's one minor problem: Accuracy. If you have a whole song playing at the same tempo, the way you calculate musical time internally quickly starts to matter. If you do the easiest thing that works... in the closure: float position; float tempo; in process(): while(samples left) { switch(event type) { case XAP_A_POSITION_CHANGE: position = event->value; break; case XAP_A_TEMPO_CHANGE: tempo = event->value; break; } for(fragment frames) { ...process audio... position += tempo; } } ...you'll probably drift off pretty soon. *Very* soon! Position should definitely be double, or the actual resolution of tempo in the addition will approach 0 bits as the difference in exponents approaches 24 bits. That means your plugin stops in it's tracks within some 6 minutes from start-of-song at 6000 ticks/second. (Hope I got the maths right, to within an order of magnitude, at least. :-) Use double for the events and the internal variables, and you'll be "fine" - but it still makes me nervous! I can see two solutions: 1) The sender must send "spontaneous" position change events every now and then. 2) Plugins that care about musical time must resync once per buffer by asking the host about the current musical time. However, 2 does not work if there's more than one timeline in the net, since then, the only thing that can tie anything to the right timeline is the events passed to a Channel. A plugin that cares about musical time can and should handle one timeline per Channel. That leaves 1 - and it doesn't seem too bad, performance wise. It's not terribly important when these extra resync events are sent; just that they're sent "often enough" for everyone to stay within the same half audio sample or better. Position and tempo changes would effectively be controls (although changed through specific events), and thus could be handled as such. Plugins that care about musical time would have a "fake" Control input "TIMELINE" or something on relevant Control Ports, and sequencers, timeline plugins and the like would obviously care to send the corresponding events only if there is somewhere to send them. [...] Interesting idea, though. However, it won't work unless the host is the one and only timeline manager, and there is only one timeline. That breaks down if there is a tempo change before that event is to be delivered. Maybe not a major issue, but it may matter a lot to some plugins in some situations. Right. Yes, definitely. Well, that would be possibly, but I don't really expect your average synth or sampler to handle that well... It's in fact impossible to handle it *correctly*, unless plugins can scan back an get events that are before the current position, despite them being skipped. However, the real reason why you want access to absolute musical time is that you need it to lock to the timeline. You can *sync* using just tempo, but that's not sufficient if you want to implement beat synchronized effects. //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Saturday 14 December 2002 07.05, Tim Hockin wrote: Yes. Yes - except that the host can't know which timeline you're asking about... (Irrelevant to VST, since you simply cannot have more than one timeline, unless you split the not in partitions belonging to different timelines. There's no way to have musical time related input from two timelines to one plugin.) Right. Yes. Exactly. That's why we use the alternative logic: The current tempo is whatever the value of the tempo control is in the timeline at the current position. We don't care whether musical time is stopped, skipping or whatever; tempo is just a control value. Could be any handy unit, as long as it's tied to a timeline, rather than free running time. I definitely vote for 1 PPQN (ie one tick per beat), which is what VST is using. No need to throw PPNQ resolution in the mix when we're dealing with floating point anyway! Pulses Per Quarter Note. Let's simply set that to one, and forget about it. We don't want plugins to keep track of some arbitrary conversion factor, that is in fact entirely irrelevant. //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Saturday 14 December 2002 16.20, Frank van de Pol wrote: But you queue ahead, and there are no "slices" or similar, are there? Since there are no slices, and thus no unit for a fixed latency, you *must* queue ahead - or you might as well use the rawmidi interface and SCHED_FIFO. Audio plugins work with one "time slice" at a time. What has happened with the timeline during a time slice (one block) is already decided before you call the first plugin in the graph. Nothing must be allowed to change during the processing of one slice. It is in fact too late as soon as the host decides to start processing the block. Same thing as passing a block to the audio interface - when it's away, it's too late to change anything. The latency is something we have to deal with, and there is no way whatsoever to get around it. (All we can do is reduce it by hacking the kernel - and that seems to be pretty damn sufficient! :-) You *could* have someone else prequeue future events and send them back to you at the right time, but then you would need to say: * When you want the events back * What the timestamps are related to (real or timeline) And if the timestamps are related to the timeline: * What to do in case of a transport stop or skip Indeed, this is doable by all means, but it's a *sequencer* - not a basic timestamped event system. Throw a public domain implementation into the plugin SDK, so we don't have to explain over and over how to do this correctly. Yes, it should be in the public domain, so people can rip the code and adapt it to their needs. There are many answers to that third quesion... OTOH, if people want to keep their source closed, you could say they *deserve* to reimplement this all over! ;-) Exactly. This is what Layla does, for example. I have thought about whether or not plugins should know the exact sample rate, but I'm afraid it's too hard to find out the real truth to be worth the effort. You can probably ask Layla what the *current* exact sample rate is, but what's the use? Plugins will be at least a few ms ahead anyway, so there's nothing much you can do about the data that is already enqueued for playback. In short, if there is a sudden change, you're screwed, period. Exactly. I think this is all that should be relevant to any plugins that are not also drivers, or otherwise need to directly deal with the relation between "engine time" and real world time. Only if you care about it. For all practical matters, you can assume the following: input_time = (sample_count - input_latency) / sample_rate; output_time = (sample_count + output_latency) / sample_rate; If the sample rate changes at "sample_count", well it's already way too late to compensate for it, because your changes will appear on the output exactly at "output_time". Now, you *could* do something about changes in the input sample rate, if you like, but I suspect that there isn't much use for that at all. If you want to record audio at the maximum quality, I'm afraid you just have to turn that wordclock sync off for a while, and let the HDR sync the recorded material during playback. (Or you should check what's the bl**dy matter with your flaky wordclock sync source! :-) Need I point out (again?) that input_latency and output_latency are *ONLY* to be used when you deal directly with external things (MIDI drivers, audio drivers, meters and stuff in GUIs,...), where you need to know the "exact" relation between net time and real time? Other plugins - be it audio or event processors - should never touch these. Latency is already compensated for by the driver/wrapper for each respective device your data may come from, or eventually receive at. This is how VST does it, and there are no holes in it in that area, AFAICT. Yes. This is why "real" time should be completely irrelevant to plugins that talk only through Audio and Control Ports. A MIDI driver *would* have to care, because it delivers events to an external device, which has to know exactly when - in real world time - the event is to be delivered. As a normal plugin, you're in pretty much the same situation; you get an event which is to be delivered at a specific time - only the timestamp is directly or indirectly related to audio time. And obviously, in a real time situation, audio is in turn related to real time. What's "time" in here? Just map directly from sample position to tick. You must ignore real time, since the relation between that, audio time and musical time, is managed elsewhere. During the processing of one block, sample position and musical time are in a fixed relation. Sample position is in a relation to real time that is controlled by the driver and hardware, and that - for all practical matters - means that audio time *is* actual time. (With some delay that you should very rarely care about.) Applies to tempo changes as well, BTW. Either way, if you don't queue events into the uncertain future, there is no problem. If there's a tempo or position change in the middle of the current block, well, what's the problem? Just do the maths. Consider this: If you were a real time process with practically zero latency, why would you ever need to think about "unexpected" changes? The timeline would be *now* by all means, and all you have to do is check whether or not *now* is the time to perform a certain action. Whether you were just told about the action, or knew about it for a minute, is irrelevant, because *now* is the first time ever you can be absolutely sure what the right thing to do is. The same applies to block based processing, only "now" is a time slice of non-zero duration, and the output you generate will be delayed by a fixed amount of time. Yes. This is what the host managed Shadow Queues are for. We don't want plugins to sort events all the time. The host knows when there are multiple sources, so let it deal with it *efficiently* when it happens. That's *exactly* why we should not prequeue events in the event system, unless we really need it. And we don't, because we can get away with just dealing with one block at a time. Prequeueing is basically just pushing the "problem" ahead of you. Yes, that's the idea, to the extent it's applicable. Audio Ports. Event Queues. POSITION_START, POSITION_STOP, POSITION_CHANGE, TEMPO_CHANGE events. Nothing much to do with 1, though, except for both having "sample frames" as the reference time unit. Well, those would be the controls. Dunno if you could say they are strictly musical, but at the time you get them, they *do* have a definite, fixed relation to the timeline they originated from. It depends only on 3 *only* when plugins need to do more than just apply the Control changes right away. It builds on 2, and is not explicitly related to 1. (See above.) This is layered upon 4, but depends only on 4 and 2. It is implemented entirely on the host side, and Plugins only have to implement 4 (and thus 2) to get 5. For building graphs? Yes; but that's not part of the API, but rather the host SDK. (You could say that about 5 as well, but OTOH, I think it's pretty nice with standard preset file formats...! *heh* So, 5 is a part of the XAP API, and there should be a more or less complete implementation in the host SDK lib.) And then there's this Bay/Channel/Port/whatever mess... :-) Related to 1, 3 and 4, but depends on nothing - it's basically just a description of what the Plugin provides. Thanks for your comments! :-) //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' .- M A I A -------------------------------------------------. | The Multimedia Application Integration Architecture | `----------------------------> http://www.linuxdj.com/maia -' --- http://olofson.net --- http://www.reologica.se ---

On Sunday 15 December 2002 00.26, Paul Davis wrote: Well, they could be "related", I think... (Read on.) Good point. One could say that every sync source generates one of these: * timing data (tempo, sample clock,...) * positional data (song position, SMPTE,...) Positional data sort of implies that you can extract timing data as well, provided you get a stream of positional data with sufficiently accurate timing. Anyway, in that other post, I think I said there *is* a relation between all of these, but I forgot to explain why: * Audio device syncs to wordclock * Sequencer uses audio for timing (nominal sample rate assumed) Note that both are just *sync* - not lock. If you wanted to sync with a VRC, you would most probably be using SMPTE instead of wordclock - and then, it would make a *lot* more sense to sync + lock the sequencer to that, and just let the audio interface do 48 kHz, or whatever you like. Either way, the point I was trying to make still stands; by the time the host starts calling all plugins to process one block of data, *all* information the plugins will access must *static* - or you will have plugins that run in "parallel" mysteriously fall out of sync. So, the sequencer (which will obviously be one of the first plugins in the graph, if not integrated in the host) will have to figure out the SMTPE, audio, MIDI clock or whatever it's syncing or locking to for the first sample of the block, construct the timeline info for the block, and pass it to the plugins that want it. At this point, all is decided. Sync and lock has been applied, and for all practical matters, you may consider this block property of the audio hardware; you just have to do some audio processing to generate the actual data. Well, it *could*, if you assumu that one SMPTE frame corresponds to N samples... ;-) Not much point, though - you would only end up modulating the tempo of the still free running sequencer, as well as the pitch of any audio you generate. :-) This is a great explanation, IMHO. //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' --- http://olofson.net --- http://www.reologica.se ---

On Sunday 15 December 2002 13.32, Paul Davis wrote: Yep. Who said anything about samples/second? Sure, that's exactly what I'm thinking about. You're suggesting that you *must* wait for each positional "event" before you do anything at all? What I'm talking about is just what every reasonably well written sequencer does when you tell it to lock to something. Confusing what with what? You can't sync a sequencer to a VCR, and let the audio interface (for the softsynths, or whatever) run at a fixed sample rate? Seems to work really rather well if the "audio interface" is built into an external MIDI synth, that doesn't sync or lock to anything it all... Well, sorry about the bad joke... //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' --- http://olofson.net --- http://www.reologica.se ---

On Monday 16 December 2002 03.22, Paul Davis wrote: No, you're right, of course. It's just that if the information comes at regular intervals and with reasonably accurate timing, you can "guess" the relative speed of time where they come from, by looking at two or more events. If you don't get at least few hundred messages per second, what else can you do? Yes indeed - that's exactly what I meant, but I seem to be using the wrong terms all the time... :-/ [...] Well, yes. The former is positional, while the latter only affects the sample rate, or "speed". If it still sounds like I'm taking those for equivalent, I think I'd better STFU and learn english. //David Olofson - Programmer, Composer, Open Source Advocate .- The Return of Audiality! --------------------------------. | Free/Open Source Audio Engine for use in Games or Studio. | | RT and off-line synth. Scripting. Sample accurate timing. | `---------------------------> http://olofson.net/audiality -' --- http://olofson.net --- http://www.reologica.se ---