On Tue, Jul 06, 2021 at 12:37:55PM +0100, Will Godfrey wrote: This is *exactly* what I profoundly hate about the way Linux and in particular systemd based distros are going. After every update you have to scan a zillion places to check if anything has been added that needs to be opted-out. If that is possible at all. Indeed. And that becomes more and more difficult. I don't want to add to the systemd bashing chorus here - I originally liked the idea of a simple and dependable process/service supervisor. But there's one thing that the systemd advocates never made very clear, and that is that the main reason why systemd was created was to make Linux more 'corporate friendly'. In other words, to take away control from the end user, whose only remaining freedom is to opt-out, with all the work and effort that takes, of what is dictated by some central administration. And of course the distro maintainers like it, as it gives them the same power. I'll give PW its chance when the developers tell me it's ready for real life. Which will mean a session with around 15 jack clients with a total of 800 or so ports. Should run without hickups while watching a youtube movie and compiling a kernel at the same time (which I can do now without any problem). Ciao, -- FA

On Tue, Jul 06, 2021 at 03:54:22PM -0700, Fernando Lopez-Lezcano wrote: That has been my impression as well. Same here. Then we both are :-) I have been using Archlinux for around 10 years. Now evaluating Artix, which is Arch without systemd. Ciao, -- FA

On Tue, 6 Jul 2021 at 21:41, Fons Adriaensen <fons(a)linuxaudio.org> wrote: Challenge accepted!... I made a little jack client with 32 input and 32 output ports that memcpy the samples. Then I started 16 of those and linked them all in a long chain. Then I linked the input of the chain to a USB mic and the output to another USB card (it needs to do adaptive resampling to keep this going), That takes about 6 seconds to setup on my machine. I run this with a buffer size of 128 samples and 48KHz. Then I started firefox and loaded a video. Then I also started compiling all of GStreamer on all cores. Here is the screenshot: https://people.freedesktop.org/~wtay/pw-load.png Works okish, some xruns here and there and this is a stock fedora setup with extra rtprio for the user. No low latency kernel or any tuning. I had to increase the max fds to 8192. I'm sure you can eliminate more xruns with some tuning. This utterly fails with jackd on this system, it doesn't even want to start all the clients, I'm sure it's something with the config somewhere... This is probably not a representative setup but at 16+ clients and 1024+ ports we're ballpark.. It probably starts to fail more with some real processing. While testing I found a scalability bug in the feedback loop detection, which should be fixed now. It might explain startup delays with complex projects... Wim

On 07/07/21 12:37, Robin Gareus wrote: This limit was bumped in v1.9.15 to 256 clients and 2048 ports. See https://jackaudio.org/news/2020/10/15/jack2-v1915-release-and-current-statu… These can still be changed at compile time, as always.

On 7/7/21 4:00 AM, Wim Taymans wrote: Just to see if it makes a difference xrun-wise you could try rebooting with the "preempt=full" kernel option. Recent Fedora 5.12* kernels are built with HAVE_PREEMPT_DYNAMIC=y so it should be possible to change preemption from "voluntary" (the default in Fedora, pretty lame for audio work) to "full" which is much better - but not as good as an RT patched kernel. < the okish part is, well, not goodish enough because of the xruns, ha ha - but then again when running jackd you may get the same performance, depends on tuning, frames per period or "quantum" size, etc > I'll have to re-test, but when I last tested rtprio was not really doing the right thing for me when running pipewire as a jackd replacement (tested using supernova, the DSP multi-core sound synthesis engine of SuperCollider). If anyone cares to read really boring stuff about my tests (Jun 18, so maybe outdated) I am including some stuff I documented at the end of this email... maybe some of this has been fixed (my proposed solution was to drink a lot of wine[*] :-) Ah, can't wait to test with my complex ardour sessions - which were failing to load in a "reasonable" time. -- Fernando [*] actual wine, not the Windows emulator version == some rt priority testing, June 18 2021 ... Several issues... more diving into source code, including supernova... < I did file a ticket with rtkit git about the hardwired limits that do not allow for audio workstation usage, we'll see if I get any answers > This is all in xxx (Fedora 34) == install patched rtkit in which priorities and cpu usage limits have been changed to more audio dsp friendly values < NOTE: max realtime priorities in rtkit are hardwired (in the source code!!) and cannot be changed, same for scheduling ring (also hardwired in the source code to SCHED_RR, does not allow use of SCHED_FIFO), so for testing I built a patched rtkit package > == install the real jack, run jack @ priority 65 - supernova threads: all FF (SCHED_FIFO), all priority 60 which is 5 less than main jackd thread, this is the normal expected behavior when using jackd ---- $ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP 1151907 FF 60 100 0.5 SLl+ DSP Thread 0 1151907 FF 60 100 0.0 SLl+ DSP Thread 1 1151907 FF 60 100 0.0 SLl+ DSP Thread 2 ---- == install pipewire-jack, set pipewire to use the rtkit module, set priority for pipewire to be 88, set priority for jack to be 71 (in local account pipewire configuration) - supernova threads: not what it should be, first thread is the right priority but wrong scheduling class (SCHED_RR which is hardwired in rtkit), second and third are the right scheduling class (SCHED_FIFO) but wrong priority. I don't know how this is happening (some clues at the end of the email). ---- $ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP 1152815 RR 71 111 0.5 SLl+ DSP Thread 0 1152815 FF 20 60 0.0 SLl+ DSP Thread 1 1152815 FF 20 60 0.0 SLl+ DSP Thread 2 ---- supernova complains on startup: Warning: cannot raise thread priority - pipewire threads: running at the right priority and scheduling class (which is hardwired in rtkit) ---- $ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep pipe 1150363 TS - 30 0.0 S<sl pipewire 1150363 RR 88 128 0.3 Ssl pipewire 1150369 TS - 30 0.0 S<l pipewire-media- 1150369 RR 88 128 0.0 Sl pipewire-media- ---- = install pipewire-jack, set pipewire to use the rt module (NOT rtkit), set priority for pipewire to be 88, set priority for jack to be 71 - supernova threads: same results as when using the rtkit module - supposedly this is going through the standard posix rt calls instead of rtkit (wrong scheduler and priority for different threads) NOW IT GETS INTERESTING (as if it were not already)... == install new rtkit with an additional patch that removes the constraint that realtime thread scheduling is forced to include SCHED_RESET_ON_FORK (which means rt fork bombs can't happen - that is, a process that gets rt privileges gets those dropped if it forks another process). == install pipewire-jack, set pipewire to use the rt module (NOT rtkit), set priority for pipewire to be 88, set priority for jack to be 71 < even though I do not activate the rtkit module in the local configuration files I see that rtkit appears again after restarting pipewire if it was killed before the restart, but if I kill it after pipewire restarts it does NOT reappear when I start supernova, so presumably pipewire is really REALLY not using it > - supernova threads: same results as when using an rtkit with SCHED_RESET_ON_FORK, which is not what I expected at all ---- $ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP 1154308 RR 71 111 0.6 SLl+ DSP Thread 0 1154308 FF 20 60 0.0 SLl+ DSP Thread 1 1154308 FF 20 60 0.0 SLl+ DSP Thread 2 ---- = install pipewire-jack, set pipewire to use rtkit module, set priority for pipewire to be 88, set priority for jack to be 71 supernova: now this is weird, this is supposedly using rtkit but the scheduling class is fine but the priority is just 20 (not 71, not 88, nothing in pipewire is _configured_ for a priority of 20). Some clues on why this might happen below... ---- $ ps -eLo pid,class,rtprio,pri,pcpu,stat,comm --sort -rtprio | grep DSP 1154408 FF 20 60 0.7 SLl+ DSP Thread 0 1154408 FF 20 60 0.0 S<Ll+ DSP Thread 1 1154408 FF 20 60 0.0 S<Ll+ DSP Thread 2 ---- ...... So nothing really works the way it is supposed to work when pipewire-jack-* is installed and we try to use realtime scheduling with supernova. I have to keep tweaking my rtkit patch, BUT diving into the rabbit hole shows these interesting snippets in pipewire, and possible clues: ---- SPA_EXPORT int jack_client_real_time_priority (jack_client_t * client) { return 20; } SPA_EXPORT int jack_client_max_real_time_priority (jack_client_t *client) { return 20; } SPA_EXPORT int jack_acquire_real_time_scheduling (jack_native_thread_t thread, int priority) { pw_log_warn("not implemented %lu %d", thread, priority); return -ENOTSUP; } --- So THAT is (probably) where the priority 20 comes from! Nothing else that I know of has that number in my test configuration. The jack API that deals with realtime priority has hardwired values, and the jack API for acquiring realtime scheduling is just not implemented. Furthermore, in the source for the rtkit module in pipewire: ---- if (sched_setscheduler(0, policy | SCHED_RESET_ON_FORK, &sp) < 0) { pw_log_warn("could not make thread realtime: %m"); return; } ---- So it would actually seem that when NOT using rtkit, pipewire is STILL enforcing part of the rtkit policy so that forked processes cannot inherit realtime privileges. Supernova somehow triggers that problem. Maybe this could be considered a bug in supernova, maybe not. It is something that could potentially be changed in supernova, I imagine.

On Wed, Jul 07, 2021 at 01:00:21PM +0200, Wim Taymans wrote: Tried something similar: 16 instances of JackGainctl (from zita-jacktools) with 32 channels (i.e. 64 ports) each, run from a Python script. With jack2 this takes 0.5s to create the clients, and on average 0.1s to connect all of them in a chain (15 * 32 connect calls). With jack1 this fails miserably. Reason is probably that jack1 recomputes the graph for each and every connection change, even if the actual client dependencies don't change [1]. Plain unpatched kernel, with -p256, no xruns after one hour. Why on earth do you need that many kernel objects (fds) to synchronise just 16 processes ? Again something doesn't scale here... See above if you were using jackd1. [1] This was one of the many things that my rejected patch (years ago) actually fixed. IIRC the complexity of jack_connect() in jackd1 is at least O(n^2) if not O(n^3) where n is the number of existing connections - this doesn't scale. Ciao, -- FA

On 07/07/21 21:44, Fons Adriaensen wrote: Except your patch was super messy, not formatted correctly and lead to other issues, as described here: https://lists.linuxaudio.org/archives/linux-audio-user/2021-February/113821… and then Rui added a note: https://lists.linuxaudio.org/archives/linux-audio-user/2021-February/113822… You keep mentioning up that patch, but it only worked for you, it didn't for anyone else.

On Wed, Jul 07, 2021 at 09:58:52PM +0100, Filipe Coelho wrote: It was indeed a very big one, and fixed several issues: * wrong order of execution of clients, * high order polynomial complexity of several basic operations. This all depended on new data structures, and there was no way to split it up in smaller steps without introducing non-functional intermediates. It was used for years on the WFS system in Parma, until I switched to Jack2 in order to have parallel execution, as well as in the production studio of La Casa della Musica, on various systems operated by the univerities of Parma, Bologna, and Pesaro, by the RAI (Italian public radio & TV) and three private Linux-based music studios. None of those reported any problems. So stop spreading plain lies. -- FA