8 Oct
2011
8 Oct
'11
3:26 a.m.
Hi list,
We are seeing unexpected interruptions of SCHED_RR audio processing
threads, and are struggling to understand why they are happening. Does
anyone have any good tips or tools to suggest to help figure out what is
preempting or delaying realtime audio threads?
The issues are coming up with Receptor [see (*) below for an intro]
running its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor
in its "dual core mode", with three instances of Native Instruments'
Kontakt 4 in DFD + multi-core mode. Either lots of held notes, or large
patches are needed to get the spikes.
With these settings there are 5 SCHED_RR threads processing audio (on a
two-core system). 2 are from Receptor, and 1 from each instance of
Kontakt. These Kontakt "helper threads" are released to do work as
possible while the audio thread is processing.
Kontakt/DFD is using mmap to bring its files into memory. This is done
in a lower priority "DFD thread", and the mapped memory is used by the
r/t audio threads.
DFD is important because the problems don't happen without it. And the
high SCHED_RR thread count is important, because the problems also don't
happen if we reduce the count.
When the spike happens there are no:
* wine bottlenecks
* system calls
* threads blocking on each other
* page faults during audio processing
There do appear to be "involuntary context switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the scheduler
is interrupting our threads. But how do you figure out why that is
happening?
There aren't many threads in the system with higher priority. All of the
5 processing threads are SCHED_RR/76. The higher priority threads in the
system are:
* migration/0 - FIFO/99
* migration/1 - FIFO/99
* watchdog/0 - FIFO/99
* watchdog/1 - FIFO/99
* posix_cpu_timer (x2) - FIFO/99
* IRQ8 (rtc) - FIFO/99
* IRQ20 (our audio card) - FIFO 77
Could other kernel activity interrupt the audio threads? Are there
issues with memory mapping, that can block other unrelated threads? Are
there just too danged many SCHED_RR threads fighting for two cores?
Anyone have any suggestions for how to trace the scheduler, and thread
or process interruptions?
Apologies for the lengthy post, but this is a tricky subject.
Thanks for any tips or insights,
Michael Ost
Muse Research & Development
(*) Receptor (http://www.museresearch.com/products/receptor.php) is a
standalone hardware VST plugin player. It runs Windows VSTs using Linux
and Wine in a proprietary host program. We have patched Wine
(ftp://anonymous+museresearch.com@ftp.museresearch.com/rpms/1.8/wine-1.1.7-muse16.fc8.src.rpm)
to make it work better with VSTs in our environment.
8 Oct
8 Oct
4:46 a.m.
On Fri, Oct 7, 2011 at 6:25 PM, Michael Ost <most(a)museresearch.com> wrote:
We are seeing unexpected interruptions of SCHED_RR
audio processing threads,
and are struggling to understand why they are happening. Does anyone have
any good tips or tools to suggest to help figure out what is preempting or
delaying realtime audio threads?
The issues are coming up with Receptor [see (*) below for an intro] running
its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor in its "dual
core mode", with three instances of Native Instruments' Kontakt 4 in DFD +
multi-core mode. Either lots of held notes, or large patches are needed to
get the spikes.
<bitching>
You know, even if I could help you, I'm not interested in doing so. I
bought a Receptor Pro Jr. just before the Receptor 2s came out. At
the time I asked Jack Ortolani (if I recall the name correctly) if the
upcoming Receptor 2 OS would be made available for Receptor Version C
hardware, like the Receptor Pro Jr., and he told me that it definitely
would, and that it wouldn't cost me anything. It seemed totally
plausible, as a Receptor is *basically* just a souped-up PC, and the
OS is just a customized version of Fedora Linux. Based on that data,
I bought the Pro Jr. (approximately 2,000 USD), only to find out that
Muse Research decided instead to create an upgrade path for people
with Revision C hardware, with an upgrade costing almost as much as
the Receptor I had just bought:
http://www.museresearch.com/products/receptor2-upgrades.php
Several users, including myself, tried to get Muse to address the problem:
http://www.kvraudio.com/forum/viewtopic.php?t=267189
http://www.kvraudio.com/forum/viewtopic.php?t=260705
Your company insisted that the reason the OS wasn't being made freely
available is because the OS would require a SATA hard drive, and that
the SATA drive would be rendered unbootable after a certain number of
boots due a BIOS issue in the Version C hardware:
http://www.kvraudio.com/forum/viewtopic.php?p=3827915#3827915
http://www.kvraudio.com/forum/viewtopic.php?p=3927549#3927549
... which was eventually debunked by users:
http://www.kvraudio.com/forum/viewtopic.php?t=288470
http://www.kvraudio.com/forum/viewtopic.php?p=4596361#4596361
Even then, some of us wanted to have the new operating system, and
would have been happy running it off of the IDE drives that came with
our Receptors. Muse Research confirmed this wasn't a problem:
http://www.kvraudio.com/forum/viewtopic.php?p=3936241#3936241
... but failed to actually say why they wouldn't allow us to upgrade
in that fashion.
I later wrote your CEO and asked for a refund on my hardware, and he
completely ignored me.
I despise your company and its business practices.
</bitching>
--
Devin Anderson
devin (at) charityfinders (dot) com
CharityFinders - http://www.charityfinders.com/
synthclone - http://synthclone.googlecode.com/
5:08 a.m.
On 10/08/2011 03:25 AM, Michael Ost wrote:
Hi list,
We are seeing unexpected interruptions of SCHED_RR audio processing
threads, and are struggling to understand why they are happening. Does
anyone have any good tips or tools to suggest to help figure out what is
preempting or delaying realtime audio threads?
https://www.osadl.org/Realtime-Preempt-Kernel.kernel-rt.0.html#builtintools
It's a bit dated but so is your 2.6.24.3 kernel. You will need to
compile the kernel with CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y
under "Kernel hacking -> Tracers -> .." to get access to
/sys/kernel/debug/tracing/
The man page of cyclictest (8) includes some hints: see the '-b' option
on how to produce traces from the scheduler. Also check out
..linux-rt-source/Documentation/trace/histograms.txt
The issues are coming up with Receptor [see (*) below
for an intro]
running its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor
in its "dual core mode", with three instances of Native Instruments'
Kontakt 4 in DFD + multi-core mode. Either lots of held notes, or large
patches are needed to get the spikes.
With these settings there are 5 SCHED_RR threads processing audio (on a
two-core system). 2 are from Receptor, and 1 from each instance of
Kontakt. These Kontakt "helper threads" are released to do work as
possible while the audio thread is processing.
Kontakt/DFD is using mmap to bring its files into memory. This is done
in a lower priority "DFD thread", and the mapped memory is used by the
r/t audio threads.
DFD is important because the problems don't happen without it. And the
high SCHED_RR thread count is important, because the problems also don't
happen if we reduce the count.
is the DFD thread calling mlock() on the mapped memory?
maybe madvise/mlock fail because you're trying to lock more pages than
RLIMIT_MEMLOCK permits? just a guess.
Documentation/trace/mmiotrace.txt should help to find out if a process
blocks due to memory mapped i/o. The debug interface is nifty and the
time consuming part is compiling a kernel with CONFIG_MMIOTRACE=y :)
When the spike happens there are no:
* wine bottlenecks
* system calls
* threads blocking on each other
* page faults during audio processing
There do appear to be "involuntary context switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the scheduler
is interrupting our threads. But how do you figure out why that is
happening?
There aren't many threads in the system with higher priority. All of the
5 processing threads are SCHED_RR/76. The higher priority threads in the
system are:
* migration/0 - FIFO/99
* migration/1 - FIFO/99
* watchdog/0 - FIFO/99
* watchdog/1 - FIFO/99
* posix_cpu_timer (x2) - FIFO/99
* IRQ8 (rtc) - FIFO/99
* IRQ20 (our audio card) - FIFO 77
Could other kernel activity interrupt the audio threads? Are there
issues with memory mapping, that can block other unrelated threads? Are
there just too danged many SCHED_RR threads fighting for two cores?
Anyone have any suggestions for how to trace the scheduler, and thread
or process interruptions?
Apologies for the lengthy post, but this is a tricky subject.
Thanks for any tips or insights,
It would not surprise me if that is one of the many issues that got
fixed since 2.6.24. That kernel still featured the BKL [BigKernelLock]
and the problem you are describing is not too far out..
3.0.6-rt17 requires a bit more work, but 2.6.39 is very stable.
Michael Ost
Muse Research & Development
(*) Receptor (http://www.museresearch.com/products/receptor.php) is a
standalone hardware VST plugin player. It runs Windows VSTs using Linux
and Wine in a proprietary host program. We have patched Wine
(ftp://anonymous+museresearch.com@ftp.museresearch.com/rpms/1.8/wine-1.1.7-muse16.fc8.src.rpm)
to make it work better with VSTs in our environment.
_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev(a)lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev
1:03 p.m.
Michael Ost wrote:
The higher priority threads in the system are:
...
* IRQ8 (rtc) - FIFO/99
Why does this interrupt get such a high priority?
(Not that it matters as I don't expect it to be used at all ...)
Are there issues with memory mapping, that can block
other unrelated
threads?
Then you would have seen page faults.
There do appear to be "involuntary context
switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the
scheduler is interrupting our threads. But how do you figure out why
that is happening? [...] All of the 5 processing threads are
SCHED_RR/76. [...] Are there just too danged many SCHED_RR threads
fighting for two cores?
RR means Round Robin, i.e., all threads with the same priority get
an equal amount of CPU without much delay, so the scheduler has to
switch between them quite often. RR is intended for threads that must
make some progress continuously. (See "man sched_setscheduler" and
especially "man sched_rr_get_interval".)
If you want to run a thread until it has finished for now, use
SCHED_FIFO.
I assume it is your intention that all those threads have equal priority
(which means to the kernel "don't care which one of them gets executed
first").
Regards,
Clemens
10 Oct
10 Oct
7:56 p.m.
On 10/08/2011 04:02 AM, Clemens Ladisch wrote:
Yes, that was what I figured too. Thanks for the confirmation.
SCHED_RR is what we want. Our realtime audio processing (and midi input)
threads all need to be able to run. They just must not be interrupted
ever by GUI or user level threads.
I'm not totally clear on how SCHED_RR and SCHED_FIFO relate though.
Would (a) a SCHED_RR/50 thread be run ahead of a SCHED_FIFO/49 thread?
And would (b) a SCHED_RR/50 thread interrupt a running SCHED_FIFO/49 thread?
Thanks for your reply,
Michael Ost
Michael Ost wrote:
No, it isn't used. This is just a by-product of the rtirq script that we
run at startup.
Have you ever seen "migration" or "watchdog" hold the CPU for any
length
of time? I was curious about "migration" since
/proc/sys/kernel/sched_migration_cost = 500000
so I set cpu affinity on the threads, but it didn't change things.
The higher priority threads in the system are:
...
* IRQ8 (rtc) - FIFO/99
Why does this interrupt get such a high priority?
(Not that it matters as I don't expect it to be used at all ...) Are there
issues with memory mapping, that can block other unrelated
threads?
Then you would have seen page faults. There do
appear to be "involuntary context switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the
scheduler is interrupting our threads. But how do you figure out why
that is happening? [...] All of the 5 processing threads are
SCHED_RR/76. [...] Are there just too danged many SCHED_RR threads
fighting for two cores?
RR means Round Robin, i.e., all threads with the same priority get
an equal amount of CPU without much delay, so the scheduler has to
switch between them quite often. RR is intended for threads that must
make some progress continuously. (See "man sched_setscheduler" and
especially "man sched_rr_get_interval".)
If you want to run a thread until it has finished for now, use
SCHED_FIFO.
I assume it is your intention that all those threads have equal priority
(which means to the kernel "don't care which one of them gets executed
first").
8:03 p.m.
On 10/07/2011 08:07 PM, Robin Gareus wrote:
Nope, I looked into that. Nothing quite that straightfoward, unfortunately.
We have tried 2.6.33 with the same results. I'll see if we can try
something even newer.
Great info. Thanks alot!
Michael Ost
On 10/08/2011 03:25 AM, Michael Ost wrote:
Excellent!
Hi list,
We are seeing unexpected interruptions of SCHED_RR audio processing
threads, and are struggling to understand why they are happening. Does
anyone have any good tips or tools to suggest to help figure out what is
preempting or delaying realtime audio threads?
https://www.osadl.org/Realtime-Preempt-Kernel.kernel-rt.0.html#builtintools
It's a bit dated but so is your 2.6.24.3 kernel. You will need to
compile the kernel with CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y
under "Kernel hacking -> Tracers -> .." to get access to
/sys/kernel/debug/tracing/
The man page of cyclictest (8) includes some hints: see the '-b' option
on how to produce traces from the scheduler. Also check out
..linux-rt-source/Documentation/trace/histograms.txt The issues are
coming up with Receptor [see (*) below for an intro]
running its 2.6.24.3 CCRMA based kernel. The bug appears with Receptor
in its "dual core mode", with three instances of Native Instruments'
Kontakt 4 in DFD + multi-core mode. Either lots of held notes, or large
patches are needed to get the spikes.
With these settings there are 5 SCHED_RR threads processing audio (on a
two-core system). 2 are from Receptor, and 1 from each instance of
Kontakt. These Kontakt "helper threads" are released to do work as
possible while the audio thread is processing.
Kontakt/DFD is using mmap to bring its files into memory. This is done
in a lower priority "DFD thread", and the mapped memory is used by the
r/t audio threads.
DFD is important because the problems don't happen without it. And the
high SCHED_RR thread count is important, because the problems also don't
happen if we reduce the count.
is the DFD thread calling mlock() on the mapped memory?
maybe madvise/mlock fail because you're trying to lock more pages than
RLIMIT_MEMLOCK permits? just a guess. Documentation/trace/mmiotrace.txt should help to find
out if a process
blocks due to memory mapped i/o. The debug interface is nifty and the
time consuming part is compiling a kernel with CONFIG_MMIOTRACE=y :)
Again, thanks for the tip. Excellent.
When the spike
happens there are no:
* wine bottlenecks
* system calls
* threads blocking on each other
* page faults during audio processing
There do appear to be "involuntary context switches" (as reported by
getrusage) when the spikes happen. This makes it seem like the scheduler
is interrupting our threads. But how do you figure out why that is
happening?
There aren't many threads in the system with higher priority. All of the
5 processing threads are SCHED_RR/76. The higher priority threads in the
system are:
* migration/0 - FIFO/99
* migration/1 - FIFO/99
* watchdog/0 - FIFO/99
* watchdog/1 - FIFO/99
* posix_cpu_timer (x2) - FIFO/99
* IRQ8 (rtc) - FIFO/99
* IRQ20 (our audio card) - FIFO 77
Could other kernel activity interrupt the audio threads? Are there
issues with memory mapping, that can block other unrelated threads? Are
there just too danged many SCHED_RR threads fighting for two cores?
Anyone have any suggestions for how to trace the scheduler, and thread
or process interruptions?
Apologies for the lengthy post, but this is a tricky subject.
Thanks for any tips or insights,
It would not surprise me if that is one of the many issues that got
fixed since 2.6.24. That kernel still featured the BKL [BigKernelLock]
and the problem you are describing is not too far out..
3.0.6-rt17 requires a bit more work, but 2.6.39 is very stable.
9:05 p.m.
On 10/10/2011 07:55 PM, Michael Ost wrote:
I'm not totally clear on how SCHED_RR and
SCHED_FIFO relate though.
Would (a) a SCHED_RR/50 thread be run ahead of a SCHED_FIFO/49 thread?
yes.
The scheduling policy only determines the ordering within the list of
runnable processes with equal static priority.
And would (b) a SCHED_RR/50 thread interrupt a running
SCHED_FIFO/49
thread?
yes.
All scheduling is preemptive: if a process with a higher static
priority becomes ready to run, the currently running process will be
preempted and returned to the wait list.
Both texts are quotes from `man 2 sched_setscheduler`.
ciao,
robin
9:08 p.m.
On 10/10/2011 09:05 PM, Robin Gareus wrote:
On 10/10/2011 07:55 PM, Michael Ost wrote:
OOPS. I misread the question. The answer is No.
The explanation is correct:
I'm not totally clear on how SCHED_RR and
SCHED_FIFO relate though.
Would (a) a SCHED_RR/50 thread be run ahead of a SCHED_FIFO/49 thread?
yes.
The scheduling policy only determines the ordering within the list of
runnable processes with equal static priority.
And would (b) a SCHED_RR/50 thread interrupt a
running SCHED_FIFO/49
thread?
yes.
All scheduling is preemptive: if a process with a higher static
priority becomes ready to run, the currently running process will be
preempted and returned to the wait list.
Both texts are quotes from `man 2 sched_setscheduler`.
ciao,
robin
_______________________________________________
Linux-audio-dev mailing list
Linux-audio-dev(a)lists.linuxaudio.org
http://lists.linuxaudio.org/listinfo/linux-audio-dev
9:31 p.m.
On Mon, Oct 10, 2011 at 09:08:51PM +0200, Robin Gareus wrote:
OOPS. I misread the question. The answer is No.
The explanation is correct:
>
> All scheduling is preemptive: if a process with a higher static
> priority becomes ready to run, the currently running process will be
> preempted and returned to the wait list.
Now you have me confused ! Why 'no' ? Surely 50 > 49 ?
Ciao,
--
FA
And would (b) a SCHED_RR/50 thread interrupt a running
SCHED_FIFO/49
thread?
yes.
9:37 p.m.
On 10/10/2011 09:31 PM, Fons Adriaensen wrote:
On Mon, Oct 10, 2011 at 09:08:51PM +0200, Robin Gareus
wrote:
OOPS. I misread the question. The answer is No.
The explanation is correct:
>
> All scheduling is preemptive: if a process with a higher static
> priority becomes ready to run, the currently running process will be
> preempted and returned to the wait list.
Now you have me confused ! Why 'no' ? Surely 50 > 49 ?
I've already corrected this with some off-list email with Michael.
Just Forget the the last follow up mail... - I somehow s/interrupt/ be
interrupted/ in my mind when re-reading. The original email is correct.
ciao,
robin
And would (b) a SCHED_RR/50 thread interrupt a running
SCHED_FIFO/49
thread?
yes.
9:38 p.m.
On 10/07/2011 07:46 PM, Devin Anderson wrote:
> On Fri, Oct 7, 2011 at 6:25 PM, Michael Ost<most(a)museresearch.com> wrote:
>
> We are seeing unexpected interruptions of SCHED_RR audio processing threads,
<snip>
<bitching>
<snip>
Wow, a painful response. It's taken me all weekend to figure out how to
reply. :(
First, I think you are misusing this mailing list, which is a technical
forum on the art of linux audio programming, for a product issue. Flame
replies like this throw cold water on a mailing list. That will stifle
development and progress of Linux as an audio platform, which I am sure
is not what you intend to do.
Second, as an engineer I fought within the company to support an upgrade
path. But we couldn't figure out a way to provide a downloadable upgrade
that we could afford to develop and support.
The upgrade was basically from Redhat 8 to Fedora 8. Not even the fedora
project, with all its resources, offers an update with that kind of
span. And even if we were able to devise a solution, an upgrade presents
other problems in terms of support, and given that most of our installed
base is non-Linux savvy and non-technical --- we thought it would be a
nightmare either way.
We also did a fair number of experiments upgrading the old hardware to
the new OS, and we found that the performance was simply not very good,
hence our CEO decided not to offer a software upgrade path, and instead
offered a very inexpensive hardware upgrade path to get our customers
into the faster and newer hardware with the newer OS as well. So we did
try to supply a solution, and if you would like to try it for yourself,
I can probably get you a hard drive with the new software and I think
you'll find that newer, more demanding plugins, which are enabled by the
newer OS upgrade, perform really poorly on the old hardware.
I am personally sorry that you were misled and left in the cold, but the
CEO did post a letter on KVR offering low cost upgrades to anyone who
was interested.
I could speak with you offline about getting a disk from us and what is
needed to do an upgrade, if you are still interested and ready to do
some linux system tweaks. But I'm not going to continue this discussion
any further on linux-audio-developers.
- Michael Ost
10:14 p.m.
On Mon, Oct 10, 2011 at 12:38 PM, Michael Ost <most(a)museresearch.com> wrote:
That much is true. I have a lot of negative feelings about your
company and the way they handled the OS situation, and when I saw you
ask question regarding your company's product, I felt justified in
posting about my experience. Muse Research made me and others like me
very angry and frustrated when they didn't fulfill promises they had
made to their customers, and made me feel isolated and ignored when
they didn't answer or even acknowledge our questions regarding the OS
situation.
Of course, just because I felt justified doesn't mean I was, and for
that, I apologize to the LAD community, and I apologize to you, as you
are certainly not your company, and did not make the decisions that
left me with negative feelings about your company.
I'll send you a PM with the rest of my reply.
--
Devin Anderson
devin (at) charityfinders (dot) com
CharityFinders - http://www.charityfinders.com/
synthclone - http://synthclone.googlecode.com/
On 10/07/2011
07:46 PM, Devin Anderson wrote:
>
> On Fri, Oct 7, 2011 at 6:25 PM, Michael Ost<most(a)museresearch.com>
> wrote:
>
> We are seeing unexpected interruptions of SCHED_RR audio processing
> threads,
<snip>
<bitching>
<snip>
Wow, a painful response. It's taken me all weekend to figure out how to
reply. :(
First, I think you are misusing this mailing list, which is a technical
forum on the art of linux audio programming, for a product issue. 
10:37 p.m.
>> Would (a) a SCHED_RR/50 thread be run ahead
of a SCHED_FIFO/49 thread?
We have three answers, yes, no, and maybe. According to the kernel scheduler the
answer is yes. The kernel does not actually have the final say here though which is
why the answers no and maybe can be equally valid. Depending on the code involved
the SCHED_RR/50 thread might be using semaphores which can cause priority
inversion depending on who owns the semaphore.
The RR/50 can request a semaphore being held my a FIFO/48 and it will pend. The
FIFO/49 can now pre-empt the FIFO/48 and hence also pre-empt the RR/50. This
may not be what was anticipated. It is bad programming but unusually easy to do,
especially with regards to audio thread, midi threads, GUI threads and all the
other gory details of audio app programming.
I only bring it up since I read the original post to concern pre-emption of high pri
threads and there are ways this can happen depending on what you are doing.
Other people on the list can probably give you better explanation of the issue and
perhaps even explain whether it is even relevant?
Regards, nick.
10:48 p.m.
On 10/10/2011 01:37 PM, Nick Copeland wrote:
I have ruled out blocking with sync'ing primitives. That isn't happening
when I get these unusual latencies. So for my particular case the answer
is "yes". Cool.
Thanks for the reply,
Michael Ost
>> Would
(a) a SCHED_RR/50 thread be run ahead of a SCHED_FIFO/49 thread?
We have three answers, yes, no, and maybe. According to the kernel
scheduler the
answer is yes. The kernel does not actually have the final say here
though which is
why the answers no and maybe can be equally valid. Depending on the code
involved
the SCHED_RR/50 thread might be using semaphores which can cause priority
inversion depending on who owns the semaphore.
The RR/50 can request a semaphore being held my a FIFO/48 and it will
pend. The
FIFO/49 can now pre-empt the FIFO/48 and hence also pre-empt the RR/50. This
may not be what was anticipated. It is bad programming but unusually
easy to do,
especially with regards to audio thread, midi threads, GUI threads and
all the
other gory details of audio app programming.
I only bring it up since I read the original post to concern pre-emption
of high pri
threads and there are ways this can happen depending on what you are doing.
Other people on the list can probably give you better explanation of the
issue and
perhaps even explain whether it is even relevant?
11 Oct
11 Oct
12:05 a.m.
Michael Ost wrote:
Have you ever seen "migration" or
"watchdog" hold the CPU for any length of time?
This shouldn't happen.
I was curious about "migration" since
/proc/sys/kernel/sched_migration_cost = 500000
When migrating threads to another CPU (core), there is no big delay
because real-time threads have well-defined scheduling behaviour and
either interrupt the running thread immediately or go into the runnable
queue like other threads that already are on that CPU.
The reason that the cost is set so high is that the new thread will run
slower because it has to pull over its data from the other cache.
I guess I can rule out SMIs because those should happen even when there
is one thread per core.
How big are the latencies you're seeing? They are not from being
interrupted by another RR thread at the same priority (see "man
sched_rr_get_interval")?
Regards,
Clemens
2:41 a.m.
On 10/10/2011 03:04 PM, Clemens Ladisch wrote:
Michael Ost wrote:
Are these "System Management Interrupts"? I'm not familiar with them. A
google search shows that 'hwlatdetect.py' can be used to detect them. I
will run that during the spikes to see if anything turns up.
Have you ever seen "migration" or
"watchdog" hold the CPU for any length of time?
This shouldn't happen.
I was curious about "migration" since
/proc/sys/kernel/sched_migration_cost = 500000
When migrating threads to another CPU (core), there is no big delay
because real-time threads have well-defined scheduling behaviour and
either interrupt the running thread immediately or go into the runnable
queue like other threads that already are on that CPU.
The reason that the cost is set so high is that the new thread will run
slower because it has to pull over its data from the other cache.
I guess I can rule out SMIs because those should happen even when there
is one thread per core. How big are the latencies you're seeing? They are
not from being
interrupted by another RR thread at the same priority (see "man
sched_rr_get_interval")?
We are wiring up the ftrace system in the kernel and cyclictest to find
out for certain. It's looking like it could be 10s of milliseconds, when
there is only 2.9 msecs to process (44.1k/128 samples). I should be able
to get more information tomorrow.
Thanks again,
Michael Ost
10:23 a.m.
Michael Ost wrote:
Tens of millisconds looks too long for a SMI. Anyway, the fact that the
delays occur only when there is CPU contention points to the kernel
scheduler.
What is the time quantum that sched_rr_get_interval() returns for these
threads?
Regards,
Clemens
On 10/10/2011 03:04 PM, Clemens Ladisch wrote:
Yes.
I guess I can rule out SMIs because those should
happen even when there
is one thread per core.
Are these "System Management Interrupts"? How big are
the latencies you're seeing? They are not from being
interrupted by another RR thread at the same priority (see "man
sched_rr_get_interval")?
We are wiring up the ftrace system in the kernel and cyclictest to find
out for certain. It's looking like it could be 10s of milliseconds, when
there is only 2.9 msecs to process (44.1k/128 samples).
10:52 a.m.
I wrote:
What is the time quantum that sched_rr_get_interval()
returns for these
threads?
Bah, the documentation of sched_rr_get_interval() is wrong; the kernel
uses a fixed RR time quantum of 100 ms which cannot be changed (except
by changing DEF_TIMESLICE in kernel/sched.c).
This means that, when you have five RR/50 threads on one core, a thread
will run for 100 ms and then be interrupted for 400 ms. The only way
for the threads to user shorter intervals is for all of them to
cooperate and to call sched_yield() after having completed such
an interval.
Regards,
Clemens
6:33 p.m.
On 10/11/2011 01:52 AM, Clemens Ladisch wrote:
I wrote:
In our case this won't be an issue. We are really only using multiple
threads so multiple cores can be used at the same time; not to divvy up
a lengthy task, which is the normal use for threading. The fact that
there are five realtime threads in our example is not a design decision,
but only an unfortunate and unavoidable side-effect of how Receptor's
and Kontakt's multi core modes work.
All the realtime threads need to complete their work within the audio
quantum (2.9 msecs in this case) or there will be drop outs. If any of
the threads actually needed 100ms, playback will be so messed up and the
cpu will be so overloaded they would probably have to hard-power off!
We have gotten some interesting results from cyclictest and the new
ftrace system. Hopefully that will point us to some kernel latency that
we can work around or fix.
Thanks... mo
What is the time quantum that
sched_rr_get_interval() returns for these
threads?
Bah, the documentation of sched_rr_get_interval() is wrong; the kernel
uses a fixed RR time quantum of 100 ms which cannot be changed (except
by changing DEF_TIMESLICE in kernel/sched.c).
This means that, when you have five RR/50 threads on one core, a thread
will run for 100 ms and then be interrupted for 400 ms. The only way
for the threads to user shorter intervals is for all of them to
cooperate and to call sched_yield() after having completed such
an interval. 
12 Oct
12 Oct
12:10 a.m.
On Tue, Oct 11, 2011 at 5:33 PM, Michael Ost <most(a)museresearch.com> wrote:
We have gotten some interesting results from
cyclictest and the new ftrace
system. Hopefully that will point us to some kernel latency that we can work
around or fix.
it would be really great if you could share at least what you find out
about the problems, even if you don't want to share the fixes.
--p
12:25 a.m.
On 10/11/2011 03:10 PM, Paul Davis wrote:
On Tue, Oct 11, 2011 at 5:33 PM, Michael
Ost<most(a)museresearch.com> wrote:
Will do.
First things for Kontakt are in our wine. (1) You need a way to set
thread priority without hitting the wine server. And (2) you need a way
to do simple thread synchronization without hitting the wine server.
There's more going on, though, that is causing the latencies. And the
output from cyclictest/ftrace is incredibly dense! It will be a slog. :(
- mo
We have gotten some interesting results from
cyclictest and the new ftrace
system. Hopefully that will point us to some kernel latency that we can work
around or fix.
it would be really great if you could share at least what you find out
about the problems, even if you don't want to share the fixes.
13 Oct
13 Oct
1:28 a.m.
On 10/07/2011 08:07 PM, Robin Gareus wrote:
It's a bit dated but so is your 2.6.24.3 kernel. You will need to
compile the kernel with CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y
under "Kernel hacking -> Tracers -> .." to get access to
/sys/kernel/debug/tracing/
I've got some output, but am not having much luck parsing it or figuring
it out. Would you (or anyone else on the list) be willing to take a look
at it?
I'm using a 2.6.33.7 kernel, by the way.
Cheers... mo
4:20 a.m.
On 10/13/2011 01:28 AM, Michael Ost wrote:
At the output? - why not. just post it. At the problem? mmh maybe,
sounds like a lot of initial setup work.
On 10/07/2011
08:07 PM, Robin Gareus wrote:
It's a bit dated but so is your 2.6.24.3 kernel. You will need to
compile the kernel with CONFIG_PREEMPT_TRACER=y CONFIG_SCHED_TRACER=y
under "Kernel hacking -> Tracers -> .." to get access to
/sys/kernel/debug/tracing/
I've got some output, but am not having much luck parsing it or figuring
it out. Would you (or anyone else on the list) be willing to take a look
at it? I'm using a 2.6.33.7 kernel, by the way.
Cheers... mo
mount -t debugfs debugfs /sys/kernel/debug
echo wakeup_rt > /sys/kernel/debug/tracing/current_tracer
cyclictest -m -p 80 -b 200 -f -T wakeup_rt
# cyclic-test shows Thread ID -> ps -> WHO is causing the delay?
cat /sys/kernel/debug/tracing/trace
# sheds light onto the WHY
HTH,
robin
25 Oct
25 Oct
2:28 a.m.
On 10/11/2011 03:10 PM, Paul Davis wrote:
On Tue, Oct 11, 2011 at 5:33 PM, Michael
Ost<most(a)museresearch.com> wrote:
As promised, here's what I found.
Our Wine provides the following required tweaks:
* Kontakt needs to be able to get and set realtime thread priority with
Get/SetThreadPriority(THREAD_PRIORITY_TIME_CRITICAL)
* Kontakt needs to be able to call Get/SetThreadPriority() without a
trip to the wine server.
* Kontakt's helper threads need to be prevented from running while audio
processing is underway unless there is a core available for them to
process on. For our host, we solved this by running them with
SCHED_FIFO/P-1, where our audio processing threads are running at
SCHED_RR/P. So, if our host is using core1, their threads can run on
core2. But if our host is using both core1 and core2 for audio
processing, they won't wake up until processing is over.
The other crucial piece appears to be improvements to the kernel between
2.6.24 and 2.6.33. We found that even with the thread priorities fixed,
we could mess up kontakt's performance by dropping disk caches (echo 2 >
/proc/sys/vm/drop_caches). That got fixed in 2.6.33, presumably by
improvements in preemtibility.
That was three week's of work! But I'm glad to have got to the bottom of
it. And thanks to the list for helping point me in the right direction.
- Michael Ost
We have gotten some interesting results from
cyclictest and the new ftrace
system. Hopefully that will point us to some kernel latency that we can work
around or fix.
it would be really great if you could share at least what you find out
about the problems, even if you don't want to share the fixes.
4560
days inactive
4577
days old
linux-audio-dev@lists.linuxaudio.org
24 comments
7 participants
participants (7)
-
Clemens Ladisch -
Devin Anderson -
Fons Adriaensen -
Michael Ost -
Nick Copeland -
Paul Davis -
Robin Gareus