Hi everyone,
I'm Wim Taymans and I'm working on a new project called PipeWire you might
have heard about [1]. I have given some general presentations about it during
its various stages of development, some of which are online [2].
PipeWire started as a way to share arbirary multimedia, wich requires vastly
different requirements regarding format support, device and memory management
than JACK. It wasn't until I started experimenting with audio processing that
the design started to gravitate to JACK. And then some of JACKs features became
a requirement for PipeWire.
The end goal of PipeWire is to interconnect applications and devices through
a shared graph in a secure and efficient way. Some of the first applications
will be wayland screen sharing and camera sharing with access control for
sandboxed applications. It would be great if we could also use this to connect
audio apps and devices, possibly unifying the pulseaudio/JACK audio stack.
Because the general design is, what I think, now very similar to JACK, many
people have been asking me if I'm collaborating with the linux pro-audio
community on this in any way at all. I have not but I really want to change
that. In this mail I hope to start a conversation about what I'm doing and I
hope to get some help and experience from the broader professional audio
developers community on how we can make this into something useful for
everybody.
I've been looking hard at all the things that are out there, including
Wayland, JACK, LV2, CRAS, GStreamer, MFT, OMX,.. and have been trying to
combine the best ideas of these projects into PipeWire. A new plugin API was
designed for hard realtime processing of any media type. PipeWire is LGPL
licensed and depends only on a standard c library. It's currently targeting
Linux.
At the core of the PipeWire design is a graph of processing nodes with arbirary
input/output ports. Before processing begins, ports need to be configured with a
format and a set of buffers for the data. Buffer data and metadata generally
lives in memfd shared memory but can also be dmabuf or anything that can be
passed as an fd between processes. There is a lot of flexibility in doing this
setup, reusing much of the GStreamer experience there is. This all happens on
the main thread, infrequently, not very important for the actual execution of
the graph.
In the realtime thread (PipeWire currently has 1 main thread and 1 realtime data
thread), events from various sources can start push/pull operations in the
graph. For the purpose of this mail, the audio sink uses a timerfd to wake up
when the alsa buffer fill level is below a threshold. This causes the sink to
fetch a buffer from its input port queue and copy it to the alsa ringbuffer. It
then issues a pull to fetch more data from all linked peer nodes for which there
is nothing queued. These peers will then eventually push another buffer in the
sink queue to be picked up in the next pull cycle of the sink. This is somewhat
similar to the JACK async scheduling model. In the generic case, PipeWire has to
walk upstream in the graph until it finds a node that can produce something (see
below how this can be optimized).
Scheduling of nodes is, contrary to JACKs (and LADSPA and LV2) single 'process'
method, done with 2 methods: process_input and process_ouput. This is done to
support more complex plugins that need to decouple input from output and to also
support a pull model for plugins. For internal clients, we directly call the
methods, for external clients we use an eventfd and a shared ringbuffer to send
the right process command to the client.
When the external client has finished processing or need to pull, it signals
PipeWire, which then wakes up the next clients if needed. This is different from
JACK, where a client directly wakes up the peers to avoid a server context
switch. JACK can do this because the graph and all client semaphores are shared.
PipeWire can't in general for a couple of reaons: 1) you need to bring mixing of
arbitrary formats to the clients 2) sandboxed clients should not be trusted with
this information and responsability. In some cases it would probably be possible
to improve that in the future (see below).
This kind of scheduling works well for generic desktop style audio and video.
Apps can send buffers of the size of their liking. Bigger buffers means higher
latency but less frequent wakeups. The sink wakeup frequency is determined by
the smallest buffer size that needs to be mixed. There is an upper limit for the
largest amount of data that is mixed in one go to avoid having to do rewinds in
alsa and still have reasonable latency when doing volume changes or adding new
streams etc.
The idea is to make a separate part of the graph dedicated to pro-audio. This
part of the graph runs with mono 32bit float sample buffers of a fixed size and
samplerate. The nodes running in this part of the graph also need to have a
fixed input-output pattern. In this part of the graph, negotiating the format
becomes trivial. We can preallocate a fixed size buffer for each port that is
used to send/mix data between nodes. Exactly like how JACK works. In this
scenario it would be possible to bring some of the graph state to trusted
clients so that they can wake up their peers directly.
As it turns out, the generic scheduling mechanism simplifies to the JACK way of
scheduling and the option to do some optimisations (can directly start push from
the sources, bundle process_input/output calls, mixing on ports is simplified by
equal buffer sizes, ...)
There is a lot more stuff that I can talk about and a lot of things that need
to be fleshed out like latency calculations, an equivalent of JACK transport,
session management, ... But this mail is already getting long :)
I would very much like to hear your ideas, comments, flames, thoughts on this
idea. I think I'm at a stage where I can present this to a bigger audience and
have enough experience with the matter to have meaningful discussions.
PipeWire is currently still in heavy development, many things can and do
still change. I'm currently writing a replacement libjack.so[3] that runs jack
clients directly on PipeWire (mixing and complicated scheduling doesn't
work yet).
Hope to hear your comments,
Wim Taymans
[1] pipewire.org
[2] https://www.youtube.com/watch?v=6Xgx7cRoS0M
[3] https://github.com/PipeWire/pipewire-jack
Hi list, some time ago a coder was asking about
making such an app. I think it's on github.
I replied that I made one, and then I posted my
very old project on github. terminator356/polyguitsynth
It used FFTs, no windowing, but actually sort of worked.
Latency was of course fundamentally an issue.
I have always had my eye on this golden egg of a goal,
but obviously it's tough, beyond my skills and time.
So I sat down this weekend to try to modernize it.
Qt, RtAudio, and especially the DSP. I looked at aubio
but unfortunately its 'aubionotes' feature is
(currently?) monophonic.
Then I stumbled across this product,
MIDI-Guitar from Jam Origins.
I realize it's Windows product and this is Linux, but...
Wow! What the... how the...
The accuracy is astounding. The latency very low.
Go and grab it, ye original poster who requested this!
Very cheap price. I bought it. Worth every penny (I told them)!
It runs flawlessly under wine with wineASIO - and - Jack2
which some said was broken in this respect...
I read they use more than just spectral stuff.
Like AI used in speech recognition and so on.
Amazing what DSP audio and image coding can do these days.
Any thoughts on coding techniques? I've read a lot of papers!
Some say using FFTs + auto-correlation comparisons.
Some say non-negative matrix.
My head spins, but this team definitely deserves praise.
Can open source come up with something?
Cheers, Tim.
i suspect this concern is a bit belated - it probably goes back to the release
of JUCE v5 in april 2017 - i dont think anything has changed since then - quite
a fuss was made at the time like starting the very next day
the GPL is still an option but it was removed from mention on the website and
now is essentially a secret - new users need to actually download and read the
source code or download and run the binary to learn that the GPL was an option
i could point you to this discussion that tried to address the down-playing of
the GPL option and the possibility of a community fork to remove the anti-
features
https://forum.juce.com/t/should-the-community-consider-a-juce-fork/22056
Hi all,
I have a proprietary Windows application (tc electronic TonePrint
editor) running under Wine, which talks to a class-compliant* USB MIDI
device (Flashback delay pedal).
I'd like to monitor, what MIDI data the application is sending to the
device. How can I do that, short of capturing and decoding the USB
communication with wireshark?
* At least I think it is. It shows up with "amidi -l", and I can connect
with aseqdump to the port, although the _device_ itself doesn't seem to
output any MIDI when I turn the knobs.
Cheers, Chris
hi all,
all the photo's shot during the lac2018@c-base Berlin are now online:
http://www.rncbc.org/lac2018
as usual from the past years, the 3rd level/click will get you deep to
the original camera jpeg file (~4MB)
cheers
--
rncbc aka Rui Nuno Capela
Hello all,
The jacktools packages (to be presented at LAC2018) are available now at
<http://kokkinizita.linuxaudio.org/linuxaudio/downloads/index.html>
You will need (from the libraries section)
- zita-convolver
- zita-resampler
- zita-jclient
- zita-audiotools
- zita-jacktools
and install those in that order.
and of course python, numpy, matplotlib, fftw3,...
Comments and feedback on LAU or LAD.
Greetings from sunny Berlin,
--
FA
