Re: [LAD] PipeWire

Hi everyone, I'm Wim Taymans and I'm working on a new project called PipeWire you might have heard about [1]. I have given some general presentations about it during its various stages of development, some of which are online [2]. PipeWire started as a way to share arbirary multimedia, wich requires vastly different requirements regarding format support, device and memory management than JACK. It wasn't until I started experimenting with audio processing that the design started to gravitate to JACK. And then some of JACKs features became a requirement for PipeWire. The end goal of PipeWire is to interconnect applications and devices through a shared graph in a secure and efficient way. Some of the first applications will be wayland screen sharing and camera sharing with access control for sandboxed applications. It would be great if we could also use this to connect audio apps and devices, possibly unifying the pulseaudio/JACK audio stack. Because the general design is, what I think, now very similar to JACK, many people have been asking me if I'm collaborating with the linux pro-audio community on this in any way at all. I have not but I really want to change that. In this mail I hope to start a conversation about what I'm doing and I hope to get some help and experience from the broader professional audio developers community on how we can make this into something useful for everybody. I've been looking hard at all the things that are out there, including Wayland, JACK, LV2, CRAS, GStreamer, MFT, OMX,.. and have been trying to combine the best ideas of these projects into PipeWire. A new plugin API was designed for hard realtime processing of any media type. PipeWire is LGPL licensed and depends only on a standard c library. It's currently targeting Linux. At the core of the PipeWire design is a graph of processing nodes with arbirary input/output ports. Before processing begins, ports need to be configured with a format and a set of buffers for the data. Buffer data and metadata generally lives in memfd shared memory but can also be dmabuf or anything that can be passed as an fd between processes. There is a lot of flexibility in doing this setup, reusing much of the GStreamer experience there is. This all happens on the main thread, infrequently, not very important for the actual execution of the graph. In the realtime thread (PipeWire currently has 1 main thread and 1 realtime data thread), events from various sources can start push/pull operations in the graph. For the purpose of this mail, the audio sink uses a timerfd to wake up when the alsa buffer fill level is below a threshold. This causes the sink to fetch a buffer from its input port queue and copy it to the alsa ringbuffer. It then issues a pull to fetch more data from all linked peer nodes for which there is nothing queued. These peers will then eventually push another buffer in the sink queue to be picked up in the next pull cycle of the sink. This is somewhat similar to the JACK async scheduling model. In the generic case, PipeWire has to walk upstream in the graph until it finds a node that can produce something (see below how this can be optimized). Scheduling of nodes is, contrary to JACKs (and LADSPA and LV2) single 'process' method, done with 2 methods: process_input and process_ouput. This is done to support more complex plugins that need to decouple input from output and to also support a pull model for plugins. For internal clients, we directly call the methods, for external clients we use an eventfd and a shared ringbuffer to send the right process command to the client. When the external client has finished processing or need to pull, it signals PipeWire, which then wakes up the next clients if needed. This is different from JACK, where a client directly wakes up the peers to avoid a server context switch. JACK can do this because the graph and all client semaphores are shared. PipeWire can't in general for a couple of reaons: 1) you need to bring mixing of arbitrary formats to the clients 2) sandboxed clients should not be trusted with this information and responsability. In some cases it would probably be possible to improve that in the future (see below). This kind of scheduling works well for generic desktop style audio and video. Apps can send buffers of the size of their liking. Bigger buffers means higher latency but less frequent wakeups. The sink wakeup frequency is determined by the smallest buffer size that needs to be mixed. There is an upper limit for the largest amount of data that is mixed in one go to avoid having to do rewinds in alsa and still have reasonable latency when doing volume changes or adding new streams etc. The idea is to make a separate part of the graph dedicated to pro-audio. This part of the graph runs with mono 32bit float sample buffers of a fixed size and samplerate. The nodes running in this part of the graph also need to have a fixed input-output pattern. In this part of the graph, negotiating the format becomes trivial. We can preallocate a fixed size buffer for each port that is used to send/mix data between nodes. Exactly like how JACK works. In this scenario it would be possible to bring some of the graph state to trusted clients so that they can wake up their peers directly. As it turns out, the generic scheduling mechanism simplifies to the JACK way of scheduling and the option to do some optimisations (can directly start push from the sources, bundle process_input/output calls, mixing on ports is simplified by equal buffer sizes, ...) There is a lot more stuff that I can talk about and a lot of things that need to be fleshed out like latency calculations, an equivalent of JACK transport, session management, ... But this mail is already getting long :) I would very much like to hear your ideas, comments, flames, thoughts on this idea. I think I'm at a stage where I can present this to a bigger audience and have enough experience with the matter to have meaningful discussions. PipeWire is currently still in heavy development, many things can and do still change. I'm currently writing a replacement libjack.so[3] that runs jack clients directly on PipeWire (mixing and complicated scheduling doesn't work yet). Hope to hear your comments, Wim Taymans [1] pipewire.org [2] https://www.youtube.com/watch?v=6Xgx7cRoS0M [3] https://github.com/PipeWire/pipewire-jack _______________________________________________ Linux-audio-dev mailing list Linux-audio-dev(a)lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev

Hi everyone, I'm Wim Taymans and I'm working on a new project called PipeWire you might have heard about [1]. I have given some general presentations about it during its various stages of development, some of which are online [2]. PipeWire started as a way to share arbirary multimedia, wich requires vastly different requirements regarding format support, device and memory management than JACK. It wasn't until I started experimenting with audio processing that the design started to gravitate to JACK. And then some of JACKs features became a requirement for PipeWire. The end goal of PipeWire is to interconnect applications and devices through a shared graph in a secure and efficient way. Some of the first applications will be wayland screen sharing and camera sharing with access control for sandboxed applications. It would be great if we could also use this to connect audio apps and devices, possibly unifying the pulseaudio/JACK audio stack. Because the general design is, what I think, now very similar to JACK, many people have been asking me if I'm collaborating with the linux pro-audio community on this in any way at all. I have not but I really want to change that. In this mail I hope to start a conversation about what I'm doing and I hope to get some help and experience from the broader professional audio developers community on how we can make this into something useful for everybody. I've been looking hard at all the things that are out there, including Wayland, JACK, LV2, CRAS, GStreamer, MFT, OMX,.. and have been trying to combine the best ideas of these projects into PipeWire. A new plugin API was designed for hard realtime processing of any media type. PipeWire is LGPL licensed and depends only on a standard c library. It's currently targeting Linux. At the core of the PipeWire design is a graph of processing nodes with arbirary input/output ports. Before processing begins, ports need to be configured with a format and a set of buffers for the data. Buffer data and metadata generally lives in memfd shared memory but can also be dmabuf or anything that can be passed as an fd between processes. There is a lot of flexibility in doing this setup, reusing much of the GStreamer experience there is. This all happens on the main thread, infrequently, not very important for the actual execution of the graph. In the realtime thread (PipeWire currently has 1 main thread and 1 realtime data thread), events from various sources can start push/pull operations in the graph. For the purpose of this mail, the audio sink uses a timerfd to wake up when the alsa buffer fill level is below a threshold. This causes the sink to fetch a buffer from its input port queue and copy it to the alsa ringbuffer. It then issues a pull to fetch more data from all linked peer nodes for which there is nothing queued. These peers will then eventually push another buffer in the sink queue to be picked up in the next pull cycle of the sink. This is somewhat similar to the JACK async scheduling model. In the generic case, PipeWire has to walk upstream in the graph until it finds a node that can produce something (see below how this can be optimized). Scheduling of nodes is, contrary to JACKs (and LADSPA and LV2) single 'process' method, done with 2 methods: process_input and process_ouput. This is done to support more complex plugins that need to decouple input from output and to also support a pull model for plugins. For internal clients, we directly call the methods, for external clients we use an eventfd and a shared ringbuffer to send the right process command to the client. When the external client has finished processing or need to pull, it signals PipeWire, which then wakes up the next clients if needed. This is different from JACK, where a client directly wakes up the peers to avoid a server context switch. JACK can do this because the graph and all client semaphores are shared. PipeWire can't in general for a couple of reaons: 1) you need to bring mixing of arbitrary formats to the clients 2) sandboxed clients should not be trusted with this information and responsability. In some cases it would probably be possible to improve that in the future (see below). This kind of scheduling works well for generic desktop style audio and video. Apps can send buffers of the size of their liking. Bigger buffers means higher latency but less frequent wakeups. The sink wakeup frequency is determined by the smallest buffer size that needs to be mixed. There is an upper limit for the largest amount of data that is mixed in one go to avoid having to do rewinds in alsa and still have reasonable latency when doing volume changes or adding new streams etc. The idea is to make a separate part of the graph dedicated to pro-audio. This part of the graph runs with mono 32bit float sample buffers of a fixed size and samplerate. The nodes running in this part of the graph also need to have a fixed input-output pattern. In this part of the graph, negotiating the format becomes trivial. We can preallocate a fixed size buffer for each port that is used to send/mix data between nodes. Exactly like how JACK works. In this scenario it would be possible to bring some of the graph state to trusted clients so that they can wake up their peers directly. As it turns out, the generic scheduling mechanism simplifies to the JACK way of scheduling and the option to do some optimisations (can directly start push from the sources, bundle process_input/output calls, mixing on ports is simplified by equal buffer sizes, ...) There is a lot more stuff that I can talk about and a lot of things that need to be fleshed out like latency calculations, an equivalent of JACK transport, session management, ... But this mail is already getting long :) I would very much like to hear your ideas, comments, flames, thoughts on this idea. I think I'm at a stage where I can present this to a bigger audience and have enough experience with the matter to have meaningful discussions. PipeWire is currently still in heavy development, many things can and do still change. I'm currently writing a replacement libjack.so[3] that runs jack clients directly on PipeWire (mixing and complicated scheduling doesn't work yet). Hope to hear your comments, Wim Taymans [1] pipewire.org [2] https://www.youtube.com/watch?v=6Xgx7cRoS0M [3] https://github.com/PipeWire/pipewire-jack _______________________________________________ Linux-audio-dev mailing list Linux-audio-dev(a)lists.linuxaudio.org https://lists.linuxaudio.org/listinfo/linux-audio-dev