[LAD] Is Piperware a successor to Jack/Pulseaudio?

[Fernando Lopez-Lezcano]

On 7/7/21 4:00 AM, Wim Taymans wrote:
> On Tue, 6 Jul 2021 at 21:41, Fons Adriaensen <fons at linuxaudio.org> wrote:
>> I'll give PW its chance when the developers tell me it's ready for
>> real life. Which will mean a session with around 15 jack clients
>> with a total of 800 or so ports. Should run without hickups while
>> watching a youtube movie and compiling a kernel at the same time
>> (which I can do now without any problem).
> 
> Challenge accepted!... I made a little jack client with 32 input and 32 output
> ports that memcpy the samples. Then I started 16 of those and linked them
> all in a long chain.
> 
> Then I linked the input of the chain to a USB mic and the output to another
> USB card (it needs to do adaptive resampling to keep this going),
> 
> That takes about 6 seconds to setup on my machine. I run this with a buffer
> size of 128 samples and 48KHz.
> 
> Then I started firefox and loaded a video. Then I also started compiling
> all of GStreamer on all cores.
> 
> Here is the screenshot: https://people.freedesktop.org/~wtay/pw-load.png
> 
> Works okish, some xruns here and there and this is a stock fedora setup
> with extra rtprio for the user. No low latency kernel or any tuning. 

Just to see if it makes a difference xrun-wise you could try rebooting 
with the "preempt=full" kernel option. Recent Fedora 5.12* kernels are 
built with HAVE_PREEMPT_DYNAMIC=y so it should be possible to change 
preemption from "voluntary" (the default in Fedora, pretty lame for 
audio work) to "full" which is much better - but not as good as an RT 
patched kernel.

< the okish part is, well, not goodish enough because of the xruns, ha 
ha - but then again when running jackd you may get the same performance, 
depends on tuning, frames per period or "quantum" size, etc >

> I had to
> increase the max fds to 8192. I'm sure you can eliminate more xruns with
> some tuning. This utterly fails with jackd on this system, it doesn't even want
> to start all the clients, I'm sure it's something with the config somewhere...
> 
> This is probably not a representative setup but at 16+ clients and 1024+ ports
> we're ballpark.. It probably starts to fail more with some real processing.

I'll have to re-test, but when I last tested rtprio was not really doing 
the right thing for me when running pipewire as a jackd replacement 
(tested using supernova, the DSP multi-core sound synthesis engine of 
SuperCollider).

If anyone cares to read really boring stuff about my tests (Jun 18, so 
maybe outdated) I am including some stuff I documented at the end of 
this email... maybe some of this has been fixed (my proposed solution 
was to drink a lot of wine[*] :-)

> While testing I found a scalability bug in the feedback loop detection, which
> should be fixed now. It might explain startup delays with complex projects...

Ah, can't wait to test with my complex ardour sessions - which were 
failing to load in a "reasonable" time.

-- Fernando

[*] actual wine, not the Windows emulator version


== some rt priority testing, June 18 2021

...

Several issues... more diving into source code, including supernova...

< I did file a ticket with rtkit git about the hardwired limits that do 
not allow for audio workstation usage, we'll see if I get any answers >

This is all in xxx (Fedora 34)

== install patched rtkit in which priorities and cpu usage limits have 
been changed to more audio dsp friendly values

< NOTE: max realtime priorities in rtkit are hardwired (in the source 
code!!) and cannot be changed, same for scheduling ring (also hardwired 
in the source code to SCHED_RR, does not allow use of SCHED_FIFO), so 
for testing I built a patched rtkit package >

== install the real jack, run jack @ priority 65

- supernova threads: all FF (SCHED_FIFO), all priority 60 which is 5 
less than main jackd thread, this is the normal expected behavior when 
using jackd

----
$ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP
1151907 FF      60 100  0.5 SLl+ DSP Thread 0
1151907 FF      60 100  0.0 SLl+ DSP Thread 1
1151907 FF      60 100  0.0 SLl+ DSP Thread 2
----

== install pipewire-jack, set pipewire to use the rtkit module, set 
priority for pipewire to be 88, set priority for jack to be 71 (in local 
account pipewire configuration)

- supernova threads: not what it should be, first thread is the right 
priority but wrong scheduling class (SCHED_RR which is hardwired in 
rtkit), second and third are the right scheduling class (SCHED_FIFO) but 
wrong priority. I don't know how this is happening (some clues at the 
end of the email).

----
$ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP
1152815 RR      71 111  0.5 SLl+ DSP Thread 0
1152815 FF      20  60  0.0 SLl+ DSP Thread 1
1152815 FF      20  60  0.0 SLl+ DSP Thread 2
----

supernova complains on startup:
Warning: cannot raise thread priority

- pipewire threads: running at the right priority and scheduling class 
(which is hardwired in rtkit)

----
$ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep pipe
1150363 TS       -  30  0.0 S<sl pipewire
1150363 RR      88 128  0.3 Ssl  pipewire
1150369 TS       -  30  0.0 S<l  pipewire-media-
1150369 RR      88 128  0.0 Sl   pipewire-media-
----

= install pipewire-jack, set pipewire to use the rt module (NOT rtkit), 
set priority for pipewire to be 88, set priority for jack to be 71

- supernova threads: same results as when using the rtkit module - 
supposedly this is going through the standard posix rt calls instead of 
rtkit (wrong scheduler and priority for different threads)

NOW IT GETS INTERESTING (as if it were not already)...

== install new rtkit with an additional patch that removes the 
constraint that realtime thread scheduling is forced to include 
SCHED_RESET_ON_FORK (which means rt fork bombs can't happen - that is, a 
process that gets rt privileges gets those dropped if it forks another 
process).

== install pipewire-jack, set pipewire to use the rt module (NOT rtkit), 
set priority for pipewire to be 88, set priority for jack to be 71

< even though I do not activate the rtkit module in the local 
configuration files I see that rtkit appears again after restarting 
pipewire if it was killed before the restart, but if I kill it after 
pipewire restarts it does NOT reappear when I start supernova, so 
presumably pipewire is really REALLY not using it >

- supernova threads: same results as when using an rtkit with 
SCHED_RESET_ON_FORK, which is not what I expected at all

----
$ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP
1154308 RR      71 111  0.6 SLl+ DSP Thread 0
1154308 FF      20  60  0.0 SLl+ DSP Thread 1
1154308 FF      20  60  0.0 SLl+ DSP Thread 2
----

= install pipewire-jack, set pipewire to use rtkit module, set priority 
for pipewire to be 88, set priority for jack to be 71

supernova: now this is weird, this is supposedly using rtkit but the 
scheduling class is fine but the priority is just 20 (not 71, not 88, 
nothing in pipewire is _configured_ for a priority of 20). Some clues on 
why this might happen below...

----
$ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP
1154408 FF      20  60  0.7 SLl+ DSP Thread 0
1154408 FF      20  60  0.0 S<Ll+ DSP Thread 1
1154408 FF      20  60  0.0 S<Ll+ DSP Thread 2
----

......

So nothing really works the way it is supposed to work when 
pipewire-jack-* is installed and we try to use realtime scheduling with 
supernova. I have to keep tweaking my rtkit patch, BUT diving into the 
rabbit hole shows these interesting snippets in pipewire, and possible 
clues:

----
SPA_EXPORT
int jack_client_real_time_priority (jack_client_t * client)
{
     return 20;
}

SPA_EXPORT
int jack_client_max_real_time_priority (jack_client_t *client)
{
     return 20;
}

SPA_EXPORT
int jack_acquire_real_time_scheduling (jack_native_thread_t thread, int 
priority)
{
     pw_log_warn("not implemented %lu %d", thread, priority);
     return -ENOTSUP;
}
---

So THAT is (probably) where the priority 20 comes from! Nothing else 
that I know of has that number in my test configuration. The jack API 
that deals with realtime priority has hardwired values, and the jack API 
for acquiring realtime scheduling is just not implemented.

Furthermore, in the source for the rtkit module in pipewire:

----
         if (sched_setscheduler(0, policy | SCHED_RESET_ON_FORK, &sp) < 0) {
         pw_log_warn("could not make thread realtime: %m");
         return;
         }
----

So it would actually seem that when NOT using rtkit, pipewire is STILL 
enforcing part of the rtkit policy so that forked processes cannot 
inherit realtime privileges. Supernova somehow triggers that problem. 
Maybe this could be considered a bug in supernova, maybe not. It is 
something that could potentially be changed in supernova, I imagine.