[linux-audio-dev] XAP and these <MEEP> timestamps...

[Frank van de Pol]

~
Good thinking David,

with a small set of events (tempo updates and position updates) you can
maintain a real-time view of the tempo map for all plugins. This is the same
concept as used in the ALSA sequencer API.

Since the plugins require prequeueing (because of the processing in blocks,
and to compensate lag caused by the plugin or processing/transmission
delays), and this prequeueing includes the tempo/position changes you'll get
into an interesting situation where events scheduled for a certain tick and
events schedulued for a certain time (=sample) might be influenced by the
tempo/position change events. In the ALSA sequencer this is solved by
allowing events to be scheduled at either a specific time or specific tick
(using two queues).

A nice question to ask is 'what is time'. I suppose that there is a direct
correlation between time and sample frequency; but what to do with
non-constant sample frequency? (This is not a hypothetical situation, since
a sampled system which is synchronised to an external source, could be
facing variable sample rate - when slaved to a VCR for instance). I believe
the answer lies in definition of which is your time master, and use that as
such; so in case of the slowed down VCR, the notion of time will only
progress slower, without causing any trouble to the system. If no house
clock or word clock is availble, things might end up hairy...

If for offline processing the audio piece is rendered (inside the XAP
architecture), this can also be done in faster or slower than real-time
depending on cpu power (I think this is a big bonus).

mapping should be:
sample position (using declared, not effective rate) -> time -> tick

for the mapping from time to tick a simple, real-time view of the relevant
part of the tempo map is used; 
	- time of last tempo change + tempo
	- time of last position change + tick  (also used for alignment)

Since for some applications a relative time is most usefull, while for
others's a absolute position is better, this is also something to look at.
Position changes and events queued at absolute position are typically not a
good match. If events are received from multiple sources, or are received
out-of-order, they have to merged; special attention required for those
cases. Same for applications that want to make changes to prequeued events
(eg. withdraw those).

To me I get the feeling that the XAP consists on a set of APIs, each very
focused and as simple as possible. Depending on the use cases for the
application/plugin, one or more of this XAP-APIs can be used.
a quick thought brings a few to my mind; a further analysis would be
required to complete the picture:

1 a XAP api for transport of the blocks of audio data 
2 a XAP api for transport of the event data
3 a XAP api for handling the time/tempo stuff (layered upon 1 & 2)
4 a XAP api for handling musical events (notes etc.), layered upon (1, 2 & 3)
5 a XAP api for the configuration/settings
6 a XAP api for the topology

etc. etc.

just some thoughts,
Frank.



On Sat, Dec 14, 2002 at 01:06:46AM +0100, David Olofson wrote:
> On Friday 13 December 2002 22.14, Tim Hockin wrote:
> > > >  Plugins can
> > > > look at
> > > > host->ticks_per_beat.
> > >
> > > No, that can change at any time (or many times) in the block.
> >
> > well, the plugin has to know ticks-per-beat and samples-per-tick. 
> > Or rather, samples-per-beat.  If we know samples per beat (tempo)
> > we can do whatever we need, right?
> 
> Yes - as long as the song position doesn't skip, because that won't 
> (*) result in tempo events. Plugins that *lock* (rather than just 
> sync) must also be informed of any discontinuities in the timeline, 
> such as skips or loops.
> 
> (*) You *really* don't want two events with the same timestamp,
>     where the first says "tempo=-Inf" and the other says
>     "tempo=120 BPM". But that would be the closest to the correct
>     way of describing a transport skip you can get. Next is
>     "running like hell" for one sample frame, and then reverting
>     to the right tempo, but that's a *really* nasty thing to do
>     to plugins that are only concerned with tempo...
> 
> 
> > Thinking again: A plugin is really concerned with the past, and how
> > it affects the future, not the future alone.
> 
> That's a good way to explain what prequeueing is really about. :-)
> 
> 
> > plugin: "I received
> > some event that needs further processing in 1.5 beats".  If it
> > knows how many samples per beat, and it receives tempo-change
> > events, what more does it REALLY need?  We can provide ticks as a
> > higher level of granularity, but is it really needed?
> 
> No. I thought some about this earlier, but forgot to write it. This 
> is all you need to maintain a perfect (almost - read on) image of the 
> timeline:
> 
> 	Tempo changes
> 		Whenever the tempo changes, you get a sample
> 		accurate event with the new tempo.
> 
> 		Unit: Ticks/sample
> 
> 	Position changes
> 		Whenever the song position changes as a result
> 		of something other than the usual tempo pushing
> 		it forward, you get an event with the new
> 		absolute song position.
> 
> 		Unit: Ticks
> 
> 	Meter changes
> 		When PPQN (who would change *that* in the middle
> 		of a song...?), meter, etc changes, some plugins
> 		will want to know this, because it affects the
> 		way they interpret musical time. This event is
> 		probably best implemented as a notifier that
> 		gives you a pointer to a static struct owned
> 		by the sender of the event. (The sequencer, the
> 		timeline plugin or the host.) (Obviously, this
> 		means that the sender will have to maintain
> 		multiple such structs if there are mid-buffer
> 		changes or multiple changes per buffer!)
> 
> 		Unit:	*XAP_time_info
> 
> 
> That's it! That's all there is to know about the timeline. And you 
> don't have to be flooded with any of these events (unless they user 
> is abusing the timeline); you'll only get them when something 
> interesting happens.
> 
> Now, there's one minor problem: Accuracy. If you have a whole song 
> playing at the same tempo, the way you calculate musical time 
> internally quickly starts to matter. If you do the easiest thing that 
> works...
> 
> 	in the closure:
> 		float position;
> 		float tempo;
> 
> 	in process():
> 		while(samples left)
> 		{
> 			switch(event type)
> 			{
> 			  case XAP_A_POSITION_CHANGE:
> 				position = event->value;
> 				break;
> 			  case XAP_A_TEMPO_CHANGE:
> 				tempo = event->value;
> 				break;
> 			}
> 			for(fragment frames)
> 			{
> 				...process audio...
> 				position += tempo;
> 			}
> 		}
> 
> ...you'll probably drift off pretty soon. *Very* soon! Position 
> should definitely be double, or the actual resolution of tempo in the 
> addition will approach 0 bits as the difference in exponents 
> approaches 24 bits. That means your plugin stops in it's tracks 
> within some 6 minutes from start-of-song at 6000 ticks/second. (Hope 
> I got the maths right, to within an order of magnitude, at least. :-)
> 
> Use double for the events and the internal variables, and you'll be 
> "fine" - but it still makes me nervous!
> 
> I can see two solutions:
> 
> 	1) The sender must send "spontaneous" position change
> 	   events every now and then.
> 
> 	2) Plugins that care about musical time must resync
> 	   once per buffer by asking the host about the current
> 	   musical time.
> 
> However, 2 does not work if there's more than one timeline in the 
> net, since then, the only thing that can tie anything to the right 
> timeline is the events passed to a Channel. A plugin that cares about 
> musical time can and should handle one timeline per Channel.
> 
> That leaves 1 - and it doesn't seem too bad, performance wise. It's 
> not terribly important when these extra resync events are sent; just 
> that they're sent "often enough" for everyone to stay within the same 
> half audio sample or better.
> 
> Position and tempo changes would effectively be controls (although 
> changed through specific events), and thus could be handled as such. 
> Plugins that care about musical time would have a "fake" Control 
> input "TIMELINE" or something on relevant Control Ports, and 
> sequencers, timeline plugins and the like would obviously care to 
> send the corresponding events only if there is somewhere to send them.
> 
> 
> [...]
> > plugin:   host->tick_me(100, cookie);  /* alert me in 100 ticks */
> > host delivers a tick event at the right time.
> >
> > I guess I don't really like that.
> 
> Interesting idea, though. However, it won't work unless the host is 
> the one and only timeline manager, and there is only one timeline.
> 
> 
> >  I'd rather see:
> >
> > plugin recognizes the need for some action in 1/4 beat.
> > plugin knows there are 18,900 samples per beat (140bpm @ 44.1k)
> > plugin delivers a long-term event to itself for now+4725 samples
> 
> That breaks down if there is a tempo change before that event is to 
> be delivered. Maybe not a major issue, but it may matter a lot to 
> some plugins in some situations.
> 
> 
> > This should solve the issue of needing to know the passing of
> > linear musical time.  It doesn't solve the need for a plugin to
> > know about looping or transports in musical time.
> 
> Right.
> 
> 
> > Does a plugin
> > need to know this?
> 
> Yes, definitely.
> 
> 
> > Maybe useful for it to go to the middle of a
> > sample or something...
> 
> Well, that would be possibly, but I don't really expect your average 
> synth or sampler to handle that well... It's in fact impossible to 
> handle it *correctly*, unless plugins can scan back an get events 
> that are before the current position, despite them being skipped.
> 
> However, the real reason why you want access to absolute musical time 
> is that you need it to lock to the timeline. You can *sync* using 
> just tempo, but that's not sufficient if you want to implement beat 
> synchronized effects.
> 
> 
> //David Olofson - Programmer, Composer, Open Source Advocate
> 
> .- The Return of Audiality! --------------------------------.
> | Free/Open Source Audio Engine for use in Games or Studio. |
> | RT and off-line synth. Scripting. Sample accurate timing. |
> `---------------------------> http://olofson.net/audiality -'
> .- M A I A -------------------------------------------------.
> |    The Multimedia Application Integration Architecture    |
> `----------------------------> http://www.linuxdj.com/maia -'
>    --- http://olofson.net --- http://www.reologica.se ---

-- 
+---- --- -- -  -   -    - 
| Frank van de Pol                  -o)    A-L-S-A
| FvdPol at home.nl                    /\\  Sounds good!
| http://www.alsa-project.org      _\_v
| Linux - Why use Windows if we have doors available?