[LAD] PipeWire

[Wim Taymans]

On Mon, 20 Aug 2018 at 01:41, Robin Gareus <robin at gareus.org> wrote:
>
> On 02/19/2018 09:39 AM, Wim Taymans wrote:
> [...]
> > I would very much like to hear your ideas, comments, flames, thoughts on this
> > idea. I think I'm at a stage where I can present this to a bigger audience and
> > have enough experience with the matter to have meaningful discussions.
>
> Hi Wim,

Hi Robin,

Thanks for taking time to reply.

>
> I think the general lack of enthusiasm about pipewire here is because it
> does not solve any issues for linux-audio and at best does not
> introduces new ones.
>
> In the past years the most prominent question that I have received is
>
>  * How can I use all of my USB Mics with Ardour on Linux?
>  * How do I uniquely identify my many MIDI devices?
>  * Why does my audio device not have proper port-names?
>  * Why can't I re-connect my device and resume work?
>
> These questions are mostly from Mac or Windows users moving to Linux ...
> and many of them moving back to MacOS.
>
> If you try to come up with a new system (think pipewire), please copy as
> many concepts as possible from Mac's CoreAudio.
>

I heard this before. Device management on linux is still pretty bad. I have not
seriously looked at how to solve any of this yet..

>
> While it is not impossible to combine multiple devices, it is not a
> straightforward to set this up. Manging devices uniquely and handling
> temporarily missing devices is not possible on GNU/Linux AFAIK.

One of the ideas with PipeWire is to move much of the logic to set up devices
and filters to another process. We would like to have the desktops come up
with policies for what to connect when and where and implement those.

This would also make it possible to do more configuration in the desktop control
panels, like rank devices (to select a master device), setup filters
for surround sound,
bass boost, echo cancellation (the things you can configure in Windows
and MacOs).

I know there a problems with uniquely identifying devices in Linux that may not
make this 100% perfect but we should be able to get to the same state as MacOs
or Windows.

The logic for combining devices exists and works well (zita-a2j/j2a) I would
like to have built-in support for this in PipeWire as soon as 2
devices interact.
MacOS has a panel for combining devices, we need something like that too.

>
> Both pulseaudio and jack had the correct idea to present audio as a
> service to applications. The server is concerned with device(s) and
> device settings. However, both fail to abstract multiple devices, map
> their port uniquely and provide multiple apps to concurrently use those
> devices for different purposes.

Are you talking about JACK? PulseAudio pretty much has this right, no?

>
> The main issue with pulse is that it is a poll API. Also pulseaudio's
> per device, per port-latency is incorrect (if set at all).

What wrong with a poll API? To me PulseAudio has more of an event based
API.. Not sure what you mean with the latency being reported
incorrectly, latency
is dynamic and you can query it, it pretty much gives you access to the read and
write pointers of the device..

> JACK on the
> other hand is too limited: single device, fixed buffersize. jackd also
> periodically wakes ups the CPU and uses power (even if no client is
> connected).

These are the main points for objecting to JACK as a generic desktop
replacement for audio and PulseAudio takes the complete opposite approach.

To me, the ideal solution would be to keep the JACK design and remove the
above mentioned limitations.

>
> Browsing around in the pipewire source I see several potential design
> issues.
>
> In particular data format conversions: The nice part about JACK is that
> uses float as only native format. Also port-memory is shared between
> application with zero-copy.

I 100% agree, arbitrary format conversions are not practical. In PipeWire, there
are 2 scenarios:

1) exclusive access to a device. You can negotiate any format and buffer layout
directly with the device. Very handy for compressed formats, or to get the
maximum performance (games). Of course only one app can use the device
but this can be allowed in certain cases.

2) non-exclusive access. A dsp module (exclusively) connects to the device that
converts to and from the canonical format (float32 mono) and the device format.
Clients then either connect with the canonical format (jack clients) or can use
the stream API (like CoreAudio's AudioQueue) to play or record data with
conversions being handled automatically. So only conversions at the entry and
exit points of the graph, everything in between is float32.

Port memory in PipeWire is also shared between applications and is pretty much
a requirement to do anything related to video. Where JACK has 1 buffer per port
allocated in the shared memory, PipeWire can have multiple buffers per (output)
port that are all shared between the connected peer ports. The reason for
multiple buffers per port is make it possible to implement *more* zero-copy
scenarios (delays, keeping reference frames for video encoding, ..)

>
> In pipewire a port can be any data-type including vorbis and worse MIDI
> is a bolted-on sub-type on an audio port.

Well, MIDI exists as a format and this is how the format is classified currently
in the format description. I guess the concern is how this midi data
will be used
and what importance it is given in the system as a canonical data format.

>
> JACK-MIDI has in the past been criticized most because MIDI was a
> dedicated type instead of JACK providing generic event-ports.

Currently there are different ways of controlling  properties and behaviour:

 - control ports: LADSPA (and other plugins) have control ports, one float
   to configure a behaviour of the algorithms. This needs a (static) port for
   each property to control, no direct link between control ports and data
   ports, no timing information, everything needs to be a float,...
- midi ports: CC messages to control behaviour. At least you only need one
  port for controlling multiple things but you need special framing to
add timing
  info (like JACK) limited to 127 values (or 14 bits by using the +32 hack), no
  easy way to map the number to a function,...
- event ports: custom format to describe events to execute. Mostly fixes the
  MIDI problems but requires a custom format.

I think you are saying that instead of making midi ports a core data type, you
would define a custom event stream and ports to control things. At the
plugin level
this certainly makes sense. Would you not have MIDI ports? Would you convert
a midi stream to an event stream? You would still need to map events from the
midi device to the plugins.

You defined an event stream for LV2 but the use of urid requires a shared type
database between clients. How do you see this work?

>
> Another conceptual issue that I see with pipewire is that it pushes sync
> downstream (like gstreamer does), instead of sources being pulled
> upstream. This in particular will make it hard to compensate for
> latencies and align outputs.

This used to be the case a long time ago but not anymore. The graph wakes
up because there is data needed in the sink and data available in the source,
Each node in the graph is activated in turn based on when there dependencies
are finished processing. All nodes know exactly when the graph was woken up,
what sample it needs and how much data was still in the device.

>
> Implementation wise there are plenty of other issues remaining to be
> discussed, e.g. context-switches, resampling, process-graph,.. but those
> are not important at this point in time.

context-switches is a big unresolved one. I currently still go back to
the server
instead of directly waking up the downstream peers when ready. It's
(theorerically)
a matter of passing the shared memory with counters and eventfd to the client..

I know the code is not the best quality yet.. Pretty much the result
of a lot of experimentation,
I hope to improve that as things fall into place.

Wim

>
> Cheers!
> robin
>