<div dir="ltr"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:#000000">
<div style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)" class="gmail_default"></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">Greetings,
Wim. Amazing project you have there. I hope you succeed. Len has
covered lots of excellent thoughts. Here are a few more, clearly
intersecting.<br><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">First
of all, it's a great idea. I'd love to see one layer which could do
all of JACK and pulse. But the pitfalls are many :-) It's worthwhile
to remember that the ALSA people tried a lot of it, the code bits and
configuration settings are still there waiting to be used, it's just
that Pulse and JACK are doing it and more so much more reliably.<br><br></div>Second,
the newer JACK+Pulse setup with Cadence controlling it is amazing, a
joy and a simplicity. Kudos extremus (sorry, I am linguistically
challenged). It does cost a bit in JACK DSP (5% on the big BNR hard
server when I tried it), but it works very reliably.<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:#000000"><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">And third, I could certainly imagine one layer with three different kinds
of ports: MIDI (using the JACK MIDI API), Pro Audio (using the JACK
audio API), and Desktop Audio (using the Pulse API). All desktop audio
ports behave like Pulse, and are controlled using the Pulse
control APIs, and by default their data is mixed into a default Desktop
Audio hardware output. At the control system level (using JACK for all), Pulse ports look like JACK ports and can be rerouted, but the underlying layer treats them differently, decouples them from the rigid round-robin of JACK. This does not make for a simple system, because there has to be both kinds of ports for the hardware audio, and I'm sure there are a lot more complications which others will think of, and which will emerge as soon as users start trying it!<br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:arial,helvetica,sans-serif;color:rgb(0,0,0)">J.E.B.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 19, 2018 at 2:39 AM, Wim Taymans <span dir="ltr"><<a href="mailto:wim.taymans@gmail.com" target="_blank">wim.taymans@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi everyone,<br>
<br>
I'm Wim Taymans and I'm working on a new project called PipeWire you might<br>
have heard about [1]. I have given some general presentations about it during<br>
its various stages of development, some of which are online [2].<br>
<br>
PipeWire started as a way to share arbirary multimedia, wich requires vastly<br>
different requirements regarding format support, device and memory management<br>
than JACK. It wasn't until I started experimenting with audio processing that<br>
the design started to gravitate to JACK. And then some of JACKs features became<br>
a requirement for PipeWire.<br>
<br>
The end goal of PipeWire is to interconnect applications and devices through<br>
a shared graph in a secure and efficient way. Some of the first applications<br>
will be wayland screen sharing and camera sharing with access control for<br>
sandboxed applications. It would be great if we could also use this to connect<br>
audio apps and devices, possibly unifying the pulseaudio/JACK audio stack.<br>
<br>
Because the general design is, what I think, now very similar to JACK, many<br>
people have been asking me if I'm collaborating with the linux pro-audio<br>
community on this in any way at all. I have not but I really want to change<br>
that. In this mail I hope to start a conversation about what I'm doing and I<br>
hope to get some help and experience from the broader professional audio<br>
developers community on how we can make this into something useful for<br>
everybody.<br>
<br>
I've been looking hard at all the things that are out there, including<br>
Wayland, JACK, LV2, CRAS, GStreamer, MFT, OMX,.. and have been trying to<br>
combine the best ideas of these projects into PipeWire. A new plugin API was<br>
designed for hard realtime processing of any media type. PipeWire is LGPL<br>
licensed and depends only on a standard c library. It's currently targeting<br>
Linux.<br>
<br>
At the core of the PipeWire design is a graph of processing nodes with arbirary<br>
input/output ports. Before processing begins, ports need to be configured with a<br>
format and a set of buffers for the data. Buffer data and metadata generally<br>
lives in memfd shared memory but can also be dmabuf or anything that can be<br>
passed as an fd between processes. There is a lot of flexibility in doing this<br>
setup, reusing much of the GStreamer experience there is. This all happens on<br>
the main thread, infrequently, not very important for the actual execution of<br>
the graph.<br>
<br>
In the realtime thread (PipeWire currently has 1 main thread and 1 realtime data<br>
thread), events from various sources can start push/pull operations in the<br>
graph. For the purpose of this mail, the audio sink uses a timerfd to wake up<br>
when the alsa buffer fill level is below a threshold. This causes the sink to<br>
fetch a buffer from its input port queue and copy it to the alsa ringbuffer. It<br>
then issues a pull to fetch more data from all linked peer nodes for which there<br>
is nothing queued. These peers will then eventually push another buffer in the<br>
sink queue to be picked up in the next pull cycle of the sink. This is somewhat<br>
similar to the JACK async scheduling model. In the generic case, PipeWire has to<br>
walk upstream in the graph until it finds a node that can produce something (see<br>
below how this can be optimized).<br>
<br>
Scheduling of nodes is, contrary to JACKs (and LADSPA and LV2) single 'process'<br>
method, done with 2 methods: process_input and process_ouput. This is done to<br>
support more complex plugins that need to decouple input from output and to also<br>
support a pull model for plugins. For internal clients, we directly call the<br>
methods, for external clients we use an eventfd and a shared ringbuffer to send<br>
the right process command to the client.<br>
<br>
When the external client has finished processing or need to pull, it signals<br>
PipeWire, which then wakes up the next clients if needed. This is different from<br>
JACK, where a client directly wakes up the peers to avoid a server context<br>
switch. JACK can do this because the graph and all client semaphores are shared.<br>
PipeWire can't in general for a couple of reaons: 1) you need to bring mixing of<br>
arbitrary formats to the clients 2) sandboxed clients should not be trusted with<br>
this information and responsability. In some cases it would probably be possible<br>
to improve that in the future (see below).<br>
<br>
This kind of scheduling works well for generic desktop style audio and video.<br>
Apps can send buffers of the size of their liking. Bigger buffers means higher<br>
latency but less frequent wakeups. The sink wakeup frequency is determined by<br>
the smallest buffer size that needs to be mixed. There is an upper limit for the<br>
largest amount of data that is mixed in one go to avoid having to do rewinds in<br>
alsa and still have reasonable latency when doing volume changes or adding new<br>
streams etc.<br>
<br>
The idea is to make a separate part of the graph dedicated to pro-audio. This<br>
part of the graph runs with mono 32bit float sample buffers of a fixed size and<br>
samplerate. The nodes running in this part of the graph also need to have a<br>
fixed input-output pattern. In this part of the graph, negotiating the format<br>
becomes trivial. We can preallocate a fixed size buffer for each port that is<br>
used to send/mix data between nodes. Exactly like how JACK works. In this<br>
scenario it would be possible to bring some of the graph state to trusted<br>
clients so that they can wake up their peers directly.<br>
<br>
As it turns out, the generic scheduling mechanism simplifies to the JACK way of<br>
scheduling and the option to do some optimisations (can directly start push from<br>
the sources, bundle process_input/output calls, mixing on ports is simplified by<br>
equal buffer sizes, ...)<br>
<br>
There is a lot more stuff that I can talk about and a lot of things that need<br>
to be fleshed out like latency calculations, an equivalent of JACK transport,<br>
session management, ... But this mail is already getting long :)<br>
<br>
I would very much like to hear your ideas, comments, flames, thoughts on this<br>
idea. I think I'm at a stage where I can present this to a bigger audience and<br>
have enough experience with the matter to have meaningful discussions.<br>
<br>
PipeWire is currently still in heavy development, many things can and do<br>
still change. I'm currently writing a replacement libjack.so[3] that runs jack<br>
clients directly on PipeWire (mixing and complicated scheduling doesn't<br>
work yet).<br>
<br>
Hope to hear your comments,<br>
Wim Taymans<br>
<br>
<br>
[1] <a href="http://pipewire.org" rel="noreferrer" target="_blank">pipewire.org</a><br>
[2] <a href="https://www.youtube.com/watch?v=6Xgx7cRoS0M" rel="noreferrer" target="_blank">https://www.youtube.com/watch?<wbr>v=6Xgx7cRoS0M</a><br>
[3] <a href="https://github.com/PipeWire/pipewire-jack" rel="noreferrer" target="_blank">https://github.com/PipeWire/<wbr>pipewire-jack</a><br>
______________________________<wbr>_________________<br>
Linux-audio-dev mailing list<br>
<a href="mailto:Linux-audio-dev@lists.linuxaudio.org">Linux-audio-dev@lists.<wbr>linuxaudio.org</a><br>
<a href="https://lists.linuxaudio.org/listinfo/linux-audio-dev" rel="noreferrer" target="_blank">https://lists.linuxaudio.org/<wbr>listinfo/linux-audio-dev</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Jonathan E. Brickman</span></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"> </span></span><span style="color:rgb(136,136,136)"></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"><a href="http://login.jsp/?at=02e47df3-a9af-4cd9-b951-1a06d255b48f&mailto=jeb@ponderworthy.com" style="color:rgb(17,85,204)" target="_blank">jeb@ponderworthy.com</a></span></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"> (785)233-9977</span></span></font></i></div><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Hear us at <a href="http://ponderworthy.com" target="_blank">http://ponderworthy.com</a> -- CDs and MP3s <a href="http://ponderworthy.com/ad-astra/ad-astra.html" target="_blank">now available!</a></span></span></font></i></div><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Music of compassion; fire, and life!!!</span></span></font></i></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>
<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Feb 19, 2018 at 2:39 AM, Wim Taymans <span dir="ltr"><<a href="mailto:wim.taymans@gmail.com" target="_blank">wim.taymans@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
I'm Wim Taymans and I'm working on a new project called PipeWire you might<br>
have heard about [1]. I have given some general presentations about it during<br>
its various stages of development, some of which are online [2].<br>
<br>
PipeWire started as a way to share arbirary multimedia, wich requires vastly<br>
different requirements regarding format support, device and memory management<br>
than JACK. It wasn't until I started experimenting with audio processing that<br>
the design started to gravitate to JACK. And then some of JACKs features became<br>
a requirement for PipeWire.<br>
<br>
The end goal of PipeWire is to interconnect applications and devices through<br>
a shared graph in a secure and efficient way. Some of the first applications<br>
will be wayland screen sharing and camera sharing with access control for<br>
sandboxed applications. It would be great if we could also use this to connect<br>
audio apps and devices, possibly unifying the pulseaudio/JACK audio stack.<br>
<br>
Because the general design is, what I think, now very similar to JACK, many<br>
people have been asking me if I'm collaborating with the linux pro-audio<br>
community on this in any way at all. I have not but I really want to change<br>
that. In this mail I hope to start a conversation about what I'm doing and I<br>
hope to get some help and experience from the broader professional audio<br>
developers community on how we can make this into something useful for<br>
everybody.<br>
<br>
I've been looking hard at all the things that are out there, including<br>
Wayland, JACK, LV2, CRAS, GStreamer, MFT, OMX,.. and have been trying to<br>
combine the best ideas of these projects into PipeWire. A new plugin API was<br>
designed for hard realtime processing of any media type. PipeWire is LGPL<br>
licensed and depends only on a standard c library. It's currently targeting<br>
Linux.<br>
<br>
At the core of the PipeWire design is a graph of processing nodes with arbirary<br>
input/output ports. Before processing begins, ports need to be configured with a<br>
format and a set of buffers for the data. Buffer data and metadata generally<br>
lives in memfd shared memory but can also be dmabuf or anything that can be<br>
passed as an fd between processes. There is a lot of flexibility in doing this<br>
setup, reusing much of the GStreamer experience there is. This all happens on<br>
the main thread, infrequently, not very important for the actual execution of<br>
the graph.<br>
<br>
In the realtime thread (PipeWire currently has 1 main thread and 1 realtime data<br>
thread), events from various sources can start push/pull operations in the<br>
graph. For the purpose of this mail, the audio sink uses a timerfd to wake up<br>
when the alsa buffer fill level is below a threshold. This causes the sink to<br>
fetch a buffer from its input port queue and copy it to the alsa ringbuffer. It<br>
then issues a pull to fetch more data from all linked peer nodes for which there<br>
is nothing queued. These peers will then eventually push another buffer in the<br>
sink queue to be picked up in the next pull cycle of the sink. This is somewhat<br>
similar to the JACK async scheduling model. In the generic case, PipeWire has to<br>
walk upstream in the graph until it finds a node that can produce something (see<br>
below how this can be optimized).<br>
<br>
Scheduling of nodes is, contrary to JACKs (and LADSPA and LV2) single 'process'<br>
method, done with 2 methods: process_input and process_ouput. This is done to<br>
support more complex plugins that need to decouple input from output and to also<br>
support a pull model for plugins. For internal clients, we directly call the<br>
methods, for external clients we use an eventfd and a shared ringbuffer to send<br>
the right process command to the client.<br>
<br>
When the external client has finished processing or need to pull, it signals<br>
PipeWire, which then wakes up the next clients if needed. This is different from<br>
JACK, where a client directly wakes up the peers to avoid a server context<br>
switch. JACK can do this because the graph and all client semaphores are shared.<br>
PipeWire can't in general for a couple of reaons: 1) you need to bring mixing of<br>
arbitrary formats to the clients 2) sandboxed clients should not be trusted with<br>
this information and responsability. In some cases it would probably be possible<br>
to improve that in the future (see below).<br>
<br>
This kind of scheduling works well for generic desktop style audio and video.<br>
Apps can send buffers of the size of their liking. Bigger buffers means higher<br>
latency but less frequent wakeups. The sink wakeup frequency is determined by<br>
the smallest buffer size that needs to be mixed. There is an upper limit for the<br>
largest amount of data that is mixed in one go to avoid having to do rewinds in<br>
alsa and still have reasonable latency when doing volume changes or adding new<br>
streams etc.<br>
<br>
The idea is to make a separate part of the graph dedicated to pro-audio. This<br>
part of the graph runs with mono 32bit float sample buffers of a fixed size and<br>
samplerate. The nodes running in this part of the graph also need to have a<br>
fixed input-output pattern. In this part of the graph, negotiating the format<br>
becomes trivial. We can preallocate a fixed size buffer for each port that is<br>
used to send/mix data between nodes. Exactly like how JACK works. In this<br>
scenario it would be possible to bring some of the graph state to trusted<br>
clients so that they can wake up their peers directly.<br>
<br>
As it turns out, the generic scheduling mechanism simplifies to the JACK way of<br>
scheduling and the option to do some optimisations (can directly start push from<br>
the sources, bundle process_input/output calls, mixing on ports is simplified by<br>
equal buffer sizes, ...)<br>
<br>
There is a lot more stuff that I can talk about and a lot of things that need<br>
to be fleshed out like latency calculations, an equivalent of JACK transport,<br>
session management, ... But this mail is already getting long :)<br>
<br>
I would very much like to hear your ideas, comments, flames, thoughts on this<br>
idea. I think I'm at a stage where I can present this to a bigger audience and<br>
have enough experience with the matter to have meaningful discussions.<br>
<br>
PipeWire is currently still in heavy development, many things can and do<br>
still change. I'm currently writing a replacement libjack.so[3] that runs jack<br>
clients directly on PipeWire (mixing and complicated scheduling doesn't<br>
work yet).<br>
<br>
Hope to hear your comments,<br>
Wim Taymans<br>
<br>
<br>
[1] <a href="http://pipewire.org" rel="noreferrer" target="_blank">pipewire.org</a><br>
[2] <a href="https://www.youtube.com/watch?v=6Xgx7cRoS0M" rel="noreferrer" target="_blank">https://www.youtube.com/watch?<wbr>v=6Xgx7cRoS0M</a><br>
[3] <a href="https://github.com/PipeWire/pipewire-jack" rel="noreferrer" target="_blank">https://github.com/PipeWire/<wbr>pipewire-jack</a><br>
______________________________<wbr>_________________<br>
Linux-audio-dev mailing list<br>
<a href="mailto:Linux-audio-dev@lists.linuxaudio.org">Linux-audio-dev@lists.<wbr>linuxaudio.org</a><br>
<a href="https://lists.linuxaudio.org/listinfo/linux-audio-dev" rel="noreferrer" target="_blank">https://lists.linuxaudio.org/<wbr>listinfo/linux-audio-dev</a><br>
</blockquote></div><br><br clear="all"><br>-- <br><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Jonathan E. Brickman</span></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"> </span></span><span style="color:rgb(136,136,136)"></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"><a href="http://login.jsp/?at=02e47df3-a9af-4cd9-b951-1a06d255b48f&mailto=jeb@ponderworthy.com" style="color:rgb(17,85,204)" target="_blank">jeb@ponderworthy.com</a></span></span><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)"> (785)233-9977</span></span></font></i></div><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Hear us at <a href="http://ponderworthy.com" target="_blank">http://ponderworthy.com</a> -- CDs and MP3s <a href="http://ponderworthy.com/ad-astra/ad-astra.html" target="_blank">now available!</a></span></span></font></i></div><div><i><font face="arial, helvetica, sans-serif" size="2"><span style="color:rgb(136,136,136)"><span color="#993300" style="color:rgb(153,51,0)">Music of compassion; fire, and life!!!</span></span></font></i></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>
</div>