[linux-audio-dev] XAP spec - early scribbles

[Tim Hockin]

All - here are some early scribbles of a XAP spec.  I took the doc that I
wrote as an overview and started formalizing it.  Please read it over and
pooint out the places where it is missing stuff?  The Table of Contents
needs a lot more meat.  Once I have that, I can write a lot or a little on
each subject.

Again - EARLY SCRIBBLES :)  That said, the ToC, content, organization,
structure, and spelling are all up for abuse.

Have at it


The XAP Audio Plugin API
Specification
$Id: xap_overview.txt,v 1.1 2003/01/15 10:55:50 thockin Exp $

0.0 Meta
  0.1 Guilty Parties
  0.2 Goals
  0.3 Terminology
  0.4 Conventions

1.0 Overview (introduce ideas, no details)
  1.1 Controls
  1.2 Events
  1.3 Channels
  1.4 Ports

2.0 The Descriptor
  2.1 Meta-Data
  2.2 State Changes
    2.2.1 Create
    2.2.2 Destroy
    2.2.3 Activate
    2.2.4 Deactivate
  2.3 Channels
  2.4 Port Setup
  2.5 Controls and EventQueues
  2.6 Processing Audio
  2.7 Errorcodes

Host struct
  - Threading
Control struct
 - RT
 - we have SYS controls
Port struct
Events
 - ramping
 -EVQ and Events
Channels
Ports
Instruments and Voices
   We have Voice Controls
Special controls
 - sys controls
 - MIDI controls
Tempo and Meter
Sequencer Controls
Macros

0.1 Guilty Parties

  XAP is a combined effort of many people on the linux-audio-dev
  (linux-audio-dev at music.columbia.edu) mailing list.  The discussion is
  open, and anyone interested is welcome to join in.

0.2 Goals

   The main goal of this project is to provide an API that is full-featured
   enough to be the primary plugin system for audio creation and playback
   applications, while remaining as simple, lightweight, and self-contained
   as possible.  The focus of this API is flexibility tempered with
   simplicity.

0.3 Terminology

  In order to read this document without having your head spin, you should
  probably understand the following terms, first.

  * Plugin:
	A chunk of code, loaded or not, that implements this API (e.g. a .so
	file or a running instance).
  * Host
	The program responsible for loading and controlling Plugins.
  * Instrument/Source:
	An instance of a Plugin that supports the instrument API and is used
	to generate audio signals.  Many Instruments will implement audio
	output but not input, though they may support both and be used an an
	Effect, too.
  * Effect:
	An instance of a Plugin that supports both audio input and output.
  * Output/Sink:
	An instance of a Plugin that can act as a terminator for a chain of
	Plugins.  Many Outputs will will support audio input but not output,
	though they may support both and be used as an Effect, too.
  * Voice:
	A playing sound within an Instrument.  Instruments may have multiple
	Voices, or only one Voice.  A Voice may be silent but still active.
  * Event:
	A time-stamped notification of some change of something.
  * Control:
	A knob, button, slider, or virtual thing that modifies behavior of
	the Plugin.  Controls can be master (e.g. master volume),
	per-Channel (e.g. channel pressure) or per-Voice (e.g. aftertouch).
  * Port:
	An audio input or output.
  * EventQueue
	A control input or output. Plugins may internally have as many
	EventQueues as they deem necessary. The Host will ask the
	Plugin for the EventQueue for each Control.
	FIXME: what is the full list of things that have a queue?
	     Controls, Plugin(master), each Channel?

  * VVID
	Virtual Voice ID. Part of a system that allows sequencers and
	the like to control synth Voices without having detailed
	knowledge of how the synth manages Voices.
  * Channel
	A grouping of Controls and Ports, similar to MIDI Channels.
  * Tick
	The unit of musical time, used with the tempo and
	meter interfaces. The unit is decided by the
	maintainer of the timeline, in order to keep musical
	time calculations exact as far as possible.
  * Cue Point
	A virtual marker on the musical time line, marking a
	position to which the plugin should be able to jump
	at any time, without delay. This is used by hard disk
	recorders, and other plugins that may need to perform
	time consuming and/or nondeterministic processing as
	a result of timeline jumps.

0.4 Conventions

  //FIXME: datatypes (XAP_foo) and documentation style

1.0 Overview

  XAP Plugins live in shared object files.  A shared object file holds
  one or more plugin descriptors, accessed by index.  Each descriptor holds
  all the information about a single plugin - it's identification,
  meta-data, capabilities, controls, and access methods.  The Plugin
  descriptors are retrieved by an exported function in each shared object.

  XAP plugins are always in one of two states - ACTIVE or IDLE.  IDLE
  Plugins are not capable of processing audio, and must be activated.
  Plugins will spend most of their time in the ACTIVE state.  After loading
  Plugins, the Host instantiates them, establishes Port and EventQueue
  connections, and activates them.  Once ACTIVE, a Plugin expects to be run
  repeatedly on small blocks of data.  When audio processing is done, the
  Host can deactivate the Plugin.

  XAP is designed to be used in realtime scenarios.  XAP plugins specify
  their realtime capabilities, and Hosts can allow or disallow operations
  based on that information.

  All XAP audio data is processed in 32-bit floating point form.  Values are
  normalized between -1.0 and 1.0, with 0.0 being silence.

1.1 Controls

  XAP uses the idea of Controls as abstract carriers of Plugin parameters
  and other information.  Controls can represent things like knobs and
  buttons, but they can also represent things like filenames, MIDI
  aftertouch or channel pressure.  Not only do they represent audio
  parameters, but they can represent chunks of system information, such as
  tempo or transport position.

  Like audio hardware, knobs and other controls can be global to a Plugin.
  However, XAP also allows Instrument Plugins to provide per-Voice controls.

  Controls come in a few datatype flavors, and can have min/max limits and
  default values, as well as hints to the host about what they are,
  semantically.  Hosts can use the hints to automatically connect things,
  where appropriate.

  Controls get their data via Events.  Events can immediately set the value
  of a Control, or they can establish a ramp - a target and duration for the
  Plugin to change the control more smoothly.

1.2 Events

  Almost everything that happens during the ACTIVE state is communicated via
  Events.  The Host can send Events to Plugins, Plugins can send Events to
  the Host, and Plugins can send Events to other Plugins (if they are so
  connected).

  All Events are timestamped.  That means that any Control change, or any
  other Event is sample-accurate.  XAP hosts have a running timer which
  counts sample-frames - this is what the timestamp is based on.

  Events are passed to Plugins on EventQueues.  This allows a Plugin to
  receive any number of Events with a minimal per-Event overhead.

1.3 Channels

  //FIXME:

1.4 Ports

  This is an audio API, so it wouldn't be complete without some mechanism to
  transport audio Data.  This is a Port.  Each Port carries a single stream
  of mono audio data.

  Plugins may allow the Host to disable Ports.  If a Port is disabled, the
  Plugin will not read from or write to it.  If a Port is not disabled, it
  must be connected by the host to a valid buffer.

2.0 The Plugin Descriptor

  The Plugin descriptor is a static(*) data structure provided by the Plugin
  to describe what it can do.  Descriptors are retrieved by calling the
  xap_descriptor() function of a XAP shared object.  This function is called
  repeatedly with an index parameter, starting at 0 and incrementing by one
  on each call.  The function returns the Plugin descriptor for each index,
  up to the number of Plugins in the shared object file, at which time it
  returns NULL.  Plugin indices are always sequential, and once a NULL is
  returned, the host can assume there are no more Plugin descriptors to be
  queried.

  (*) The descriptor for wrapper Plugins may change upon loading of a
  wrapped Plugin.  This is a special case, and the Host must be aware of it.
  //FIXME: how?

2.1 Meta-Data

  The Plugin descriptor provides several fields for Plugin meta-data.  The
  fields are as follows:

  id_code:	the vendor and product encoding
  api_code:	the XAP API version code of this Plugin
  ver_code:	the Plugin version code - used to identify Plugin data
  flags:	Plugin-global flags
  label:	a short, unique Plugin identifier
  name:		the user-friendly Plugin name string
  version:	the version string
  author:	the author string
  copyright:	the copyright string
  license:	the license string
  url:		the URL string
  notes:	notes about the Plugin

2.2 State Changes

  As mentioned above, Plugins exist in one of two states - ACTIVE and
  IDLE (well, three if you count non-existance as a state).  The Plugin
  descriptor holds the methods to create and destroy instances of Plugins,
  and to change their state.

2.2.1 Create

  A Plugin is instantiated via it's descriptor's create() method.  This
  method receives two key pieces of information, which the Plugin will use
  throughout it's lifetime.  The first is a pointer to the Host structure
  (see below), and the second is the Host sample rate.  If the Host wants to
  change either of these, all Plugins must be re-created.  This is where the
  Plugin's internal structures can be allocated and initialized.  There is
  no required set of supported sample rates, but Plugins should support the
  common sample rates (44100, 48000, 96000) to be generally useful.  If the
  Plugin does not support the specified sample rate, this method should
  fail.  Hosts should always check that all Plugins support the desired
  sample rate.

  Once created, the Plugin instance is in the IDLE state.

2.2.2 Destroy

  Plugins are destructed via the descriptor's destroy() method.  Plugins
  can only be destroyed when in the IDLE state.  All Plugin-allocated
  resources must be released during this method.  After this method is
  invoked, the Plugin handle is no longer valid.  This function can not
  fail.

2.2.3 Activate

  From the IDLE state, a Plugin can be changed to the ACTIVE state via the
  activate() method.  Passed to this method are two arguments which are
  valid for the duration of the ACTIVE state - quality level and
  realtime state.  The quality level is an integer between 1 and 10, with 1
  being lowest quality (fastest) and 10 being highest quality (slowest).
  Plugins may ignore this value, or may provide less than 10 discrete
  quality levels.  The realtime state is a boolean value which is boolean
  TRUE if the Plugin is in a realtime processing net or boolean FALSE if it
  is not realtime (offline).

  This method can only be called from the IDLE state.  Once ACTIVE, a Plugin
  may process audio.

2.2.4 Deactivate

  From the ACTIVE state, a Plugin can be changed to the IDLE state via the
  deactivate() method.  This method can only be called from the ACTIVE
  state.

2.3 Channels

  //FIXME:

2.4 Port Setup

  //FIXME: kinda needs Channels
  Once a Plugin is loaded, the Host must connect the audio Ports.  All Ports
  in a Plugin must be connected or disabled.  The Plugin descriptor provides
  a connect_port() method, which the host must call to connect a buffer
  pointer to a Port.  Once connected, a Port remains connected to the
  specified buffer until the Host disables it or connects it to a different
  buffer.  All Plugins are assumed to not use the same input buffers as
  output buffers, unless the Plugin flags indicate that it safely handles
  in-place processing.

  Plugins may allow the Host to disable Ports, rather than connect them.
  The Plugin descriptor provides an optional disable_port() method.  If this
  method is provided, and it returns successfully, the Host can ignore this
  port.  Once disabled, a Port remains disabled until the Host connects it,
  at which point it becomes enabled.

2.5 Controls and EventQueues

  //FIXME: kinda needs Channels
  In order to be of any use, a Plugin must provide Controls.  Control
  changes are delivered via Events.  Events are passed on EventQueues.
  Each Control has an associated EventQueue, on which Events for that
  Control are delivered.  In addition, there is an EventQueue for each
  Channel and a master Queue for the Plugin.  The Plugin can internally use
  the same EventQueue for multiple targets.

  The Host queries the Plugin for EventQueues via the get_input_queue()
  method.  In order to allow sharing of an EventQueue, the get_input_queue()
  method also returns a cookie, which is stored in each Event as it is
  delivered.  This allows the plugin to use a simpler EventQueue scheme
  internally, while still being able to sort incoming Events.

  Controls may also output Events.  The Host will set up output Controls
  with the set_output_queue() method.

2.6 Processing Audio

  A Plugin which has been activated and properly set up, may be called upon
  to process audio.  This is done through the descriptor's run() method.
  This method gets the current sample-frame timestamp as an argument.  In
  this method the Plugin is expected to examine and handle all new Events,
  and to read from or write to it's Ports.

  This method may only be called from the ACTIVE state.

2.7 Errorcodes







All this stuff needs to be integrated somewhere...

Tempo and Meter
----
  XAP uses Controls for transmitting tempo and meter information.  If a
  Plugin defines a TEMPO control, it can expect to receive tempo Events on
  that control.  The Host must define some unit of musical-time measurement (Tick),
  which represents the smallest granularity the host wants to work with.
  This is the basis for tempo and meter.  The host publishes the current count of
  Ticks/Beat via the host struct.

  Control: TEMPO
  Type: double
  Units: ticks/sec
  Range: [-inf, inf]
  Events: Hosts must send a TEMPO Event at Plugin init and when tempo
          changes.

  Control: METER
  Type: double
  Units: ticks/measure
  Range: [0.0, inf]
  Events: Hosts must send a METER Event at Plugin init and when meter
          changes.  Hosts should send a METER Event periodically, such as
          every measure or once per second.

  Control: METERBASE
  Type: double
  Units: beats/whole-note
  Range: [1.0, inf]
  Events: Hosts must send a METERBASE Event at Plugin init and when meter
          changes.

  This mechanism gives Plugins the ability to be aware of tempo and meter
  changes, without forcing information into plugins that don't care.  A
  Plugin can sync to various timeline events, easily.

  The Host struct also provides a mechanism to query the timestamp of the
  next Beat or Bar, and to convert timestamps into the following time formats:
    * Ticks
    * Seconds
    * SMPTE frames

Sequencer Control
----
  XAP plugins may be aware of certain sequencer events, such as transport
  changes, positional jumps., and loop-points.  These data are received on
  Controls.

  Control: POSITION
  Type: double
  Units: ticks
  Range: [0.0, inf]
  Events: Hosts must send a POSITION Event at Plugin init, when transport
          starts, and when transport jumps.  Hosts should send a POSITION
          Event periodically, such as every beat, every measure or once per
          second.

  Control: TRANSPORT
  Type: bool
  Units: on/off
  Events: Hosts must send a TRANSPORT Event at Plugin init and when
          transport state changes.

  Control: CUEPOINT
  Type: double
  Units: ticks
  Range: [0.0, inf]
  Events: Hosts must send a CUEPOINT Event when Cuepoints are added,
          changed, or removed.

  Control: SPEED
  Type: double
  Units: scalar
  Range: [-inf, inf]
  Events: Hosts must send a SPEED Event at Plugin init and when play speed
          changes.

Instruments and Voices
----
  XAP instruments can be either voiced or non-voiced.  Non-voiced
  instruments are essentially always on, and their output is controlled
  purely by controls, such as the gate control of a modular synth or the
  hand-distance of a theremin.  Non-voiced instruments are monophonic
  per-channel.

  Voiced controls are more structured.  They must handle Virtual Voice IDs
  (VVIDs).  VVIDs are unsigned 32 bit integers, which are allocated by the
  host and passed to instruments via the XAP_EV_VVID_* events.  Instruments
  may use VVIDs as a direct index into the host structure's VVID table,
  which is an array of unsigned 32 bit integers.  The data in the host VVID
  table is for use by the instrument.

  VVIDs must be allocated before use and de-allocated before re-use.
  Instruments can maintain an internal mapping between VVIDs and actual
  voices, which allows them to handle voice allocation in a purely
  abstracted manner from the host. Once allocated, the host can send events
  for a VVID until the VVID is de-allocated.

  A VVID has two states - active and inactive.  After allocation, the VVID
  is inactive.  Control events received while inactive can be assumed to set
  the control state for the activation event.  A VVID is activated via the
  VOICE control, which all instruments must provide.  Once in the active
  state, the instrument may produce sound for the voice.  A VVID is
  deactivated via the VOICE control, which puts the VVID in the inactive
  state.  It is important to note that just because a VVID has received a
  VOICE OFF event, it is not necessarily silent.  It may have a long release
  phase, which is dependant on the instrument.
  
  Control: VOICE
  Type: bool
  Units: on/off
  Events: Hosts send VOICE Events any time after a VVID has been allocated
          but before it has been deallocated.

  Some instruments will choose to only examine some controls at activation
  time (such as velocity for MIDI-like instruments) or at deactivation time
  (such as release velocity).  These controls are referred to as latched.
  Setting an init-latched control after activation may or may not set the
  value, and may or may not have any effect - this is instrument dependant.

  A VVID may be re-used.  That is to say, the host can leave a VVID
  allocated after deactivation, and re-activate it later.  This can be used
  for emulation of MIDI synths, where the VVID is akin to the MIDI pitch. 

  A VVID may be deallocated before the associated voice is done playing.
  The instrument should continue to play the voice, as no VOICE OFF event
  has been received.  Likewise, a voice may end before it has received a
  VOICE OFF (such as a drum hit).  The VVID can change to the inactive state
  and track control changes until another VOICE ON is received.

  Because VVIDs and voices have no exact correlation, instrument plugins
  have a great deal of control over their voice operations.  An Instrument
  can be mono, and cut the voice and restart for each new VOICE ON, or it
  man be massively polyphonic.  The host always alerts the plugin to the
  state of a VVID, and to changes of that state.  Once a VVID has been
  deallocated, it can be re-used.

  Plugins can assume that VVIDs are at least global per-Plugin.  The host
  will not activate the same VVID in different Channels at the same time.