9 May
2006
9 May
'06
5:56 p.m.
Lee Revell:
Subject: Re: [linux-audio-user] -rt IRQ handler
priorities (was: Re:
molnar patch)
To: Florian Paul Schmidt <mista.tapas(a)gmx.net>
Cc: Ingo Molnar <mingo(a)elte.hu>hu>, linux-audio-user(a)music.columbia.edu
Message-ID: <1147129704.9116.21.camel@mindpipe>
Content-Type: text/plain
On Tue, 2006-05-09 at 00:47 +0200, Florian Paul Schmidt wrote:
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
On Mon, 8 May 2006 21:54:53 +0200
Wolfgang Hoffmann <woho(a)woho.de> wrote:
I haven't been bitten by that. Do you also get xruns with [i suppose so,
just asking to make sure]? I haven't had as much time as before to play
around and test things, so maybe it has crept into the kernel recently
or maybe i just always had high-res timers enabled.
Not really. The problem with making the softirq timer thread high
priority is that lots of random code gets run from softirq timer
context, including rt_secret_rebuild() which is a well known latency
killer.
I browsed
over that page and didn't find any info on setting up the irq
handler priorities which is _the_ essential feature of -rt.
Apropos: on your page on -rt setup (excellent page, btw., many thanks! :-),
you suggest raising "softirq-timer/0" to prio 99, to make sleep() function
right (http://tapas.affenbande.org/?page_id=40 sleep() based/system timer).
I did so, and got strange latencies (> 40 ms) exactly once every 10 minutes,
caused by some routing-related action (rt_secret_rebuild) being run by the
softirq-timer/0 thread. Don't you get bit by that, too? Kernel is 2.6.16-rt16 My solution is to configure with
CONFIG_HIGH_RES_TIMERS=y. Then, sleep() wakes
up correctly even with softirq-timer/0 being low-priority (SCHED_FIFO 1 or
even SCHED_OTHER).
In general I find adjusting priorities of the various softirq kernel threads a
bit of secret art. I can't find much documentation about "what kernel thread
runs which job" that would help making some proper decisions here. I found my
desktop "feels" most responsive when demoting all softirq thread to
SCHED_OTHER. I did so after seeing that with a non-rt kernel, bottom-half
handler don't run SCHED_FIFO/_RR at all. So -rt now gives me robust low
latencies for jackd, and still proper desktop feeling.
Well, maybe this is getting off-topic for this list. But it seems to me
trimming priorities between kernel and userland threads is a bit like no
man's land.
I agree. Maybe Lee Revell knows more [CC'ing him]. Lee, you know
something about all these softirq threads? What exactly do they do? 
9 May
9 May
6:04 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
Steven, can you suggest a way forward? Briefly the problem is that
arbitrary latency killing code can run from the softirq timer thread,
but this thread needs to run at max RT priority for sleep(), select() &
friends to work.
I guess it hasn't been a problem yet because most RT threads want to be
awakened by some external event like an interrupt, rather than a timer.
Lee
On Tue, 2006-05-09 at 12:56 -0700, Kjetil S. Matheussen wrote:
Lee Revell:
Subject: Re: [linux-audio-user] -rt IRQ handler
priorities (was: Re:
molnar patch)
To: Florian Paul Schmidt <mista.tapas(a)gmx.net>
Cc: Ingo Molnar <mingo(a)elte.hu>hu>, linux-audio-user(a)music.columbia.edu
Message-ID: <1147129704.9116.21.camel@mindpipe>
Content-Type: text/plain
On Tue, 2006-05-09 at 00:47 +0200, Florian Paul Schmidt wrote:
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
On Mon, 8 May 2006 21:54:53 +0200
Wolfgang Hoffmann <woho(a)woho.de> wrote:
Not really. The problem with making the softirq timer thread high
priority is that lots of random code gets run from softirq timer
context, including rt_secret_rebuild() which is a well known latency
killer.
> I browsed over that page and didn't find
any info on setting up the irq
> handler priorities which is _the_ essential feature of -rt.
Apropos: on your page on -rt setup (excellent page, btw., many thanks! :-),
you suggest raising "softirq-timer/0" to prio 99, to make sleep() function
right (http://tapas.affenbande.org/?page_id=40 sleep() based/system timer).
I did so, and got strange latencies (> 40 ms) exactly once every 10 minutes,
caused by some routing-related action (rt_secret_rebuild) being run by the
softirq-timer/0 thread. Don't you get bit by that, too? Kernel is 2.6.16-rt16
I haven't been bitten by that. Do you also get xruns with [i suppose so,
just asking to make sure]? I haven't had as much time as before to play
around and test things, so maybe it has crept into the kernel recently
or maybe i just always had high-res timers enabled.
My solution is to configure with
CONFIG_HIGH_RES_TIMERS=y. Then, sleep() wakes
up correctly even with softirq-timer/0 being low-priority (SCHED_FIFO 1 or
even SCHED_OTHER).
In general I find adjusting priorities of the various softirq kernel threads a
bit of secret art. I can't find much documentation about "what kernel thread
runs which job" that would help making some proper decisions here. I found my
desktop "feels" most responsive when demoting all softirq thread to
SCHED_OTHER. I did so after seeing that with a non-rt kernel, bottom-half
handler don't run SCHED_FIFO/_RR at all. So -rt now gives me robust low
latencies for jackd, and still proper desktop feeling.
Well, maybe this is getting off-topic for this list. But it seems to me
trimming priorities between kernel and userland threads is a bit like no
man's land.
I agree. Maybe Lee Revell knows more [CC'ing him]. Lee, you know
something about all these softirq threads? What exactly do they do?
6:07 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 2006-05-09 at 12:56 -0700, Kjetil S. Matheussen wrote:
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
I think it's unusual to want a high priority SCHED_FIFO thread to be
awakened by a timer. What are you trying to do?
Lee
6:28 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 9 May 2006, Lee Revell wrote:
On Tue, 2006-05-09 at 12:56 -0700, Kjetil S.
Matheussen wrote:
Well, the reason for stumble upon this problem is when I ported/tested
the general watchdog program "das_watchdog" from 2.4 to 2.6. For some
reason, it just didn't work. But after reading the instructions for
Florians rt_watchdog program, it became clear that the softirq
thread must have priority 99.
But lets say that you have program with two SCHED_FIFO threads. One
of them is a high priority (between 2 and 98) timer that is only
used to signal other threads. The other SCHED_FIFO thread has priority
1 and needs to run realtime for some reason, and it might not even be
a good reason. However, if that other thread have to do
some hard work, it stalls your high priority timer thread for a small
time, and your sleep-based timer suddenly becomes shaky.
I don't know any real-world examples of this, but I don't find it
unlikely to already happen, because its not always obviously noticable.
For example, that cpu-hogging thread can very well be a jack client
like jamin, and that timer process might very well be muse or
rosegarden. And for all we know, the scenario drawn up above, can
be already be relevant for a large percentage of all linux audio users,
its just not obviously noticable.
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
I think it's unusual to want a high priority SCHED_FIFO thread to be
awakened by a timer. What are you trying to do?
8:33 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 2006-05-09 at 13:27 -0700, Kjetil S. Matheussen wrote:
On Tue, 9 May 2006, Lee Revell wrote:
Hmm, it sounds like a solution could be to separate timers that just
wake up a process from ones that do actual work and run them in separate
kernel threads.
Lee
On Tue, 2006-05-09 at 12:56 -0700, Kjetil S.
Matheussen wrote:
Well, the reason for stumble upon this problem is when I ported/tested
the general watchdog program "das_watchdog" from 2.4 to 2.6. For some
reason, it just didn't work. But after reading the instructions for
Florians rt_watchdog program, it became clear that the softirq
thread must have priority 99.
But lets say that you have program with two SCHED_FIFO threads. One
of them is a high priority (between 2 and 98) timer that is only
used to signal other threads. The other SCHED_FIFO thread has priority
1 and needs to run realtime for some reason, and it might not even be
a good reason. However, if that other thread have to do
some hard work, it stalls your high priority timer thread for a small
time, and your sleep-based timer suddenly becomes shaky.
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
I think it's unusual to want a high priority SCHED_FIFO thread to be
awakened by a timer. What are you trying to do?
8:45 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 9 May 2006, Lee Revell wrote:
On Tue, 2006-05-09 at 13:27 -0700, Kjetil S.
Matheussen wrote:
I don't understand why you want that. To me (which
knows about nothing about how the kernel works), the solution to
the problem is crystal clear: The softirq timer needs to have the highest
priority, and the only thing the sofirq timer threads does is to
make sure threads that should be woken up are put into some kernel
schedule queue somehow. I don't understand why the softirq timer threads
should cause any latency problems, what does it do except scheduling
waiting threads?
On Tue, 9 May 2006, Lee Revell wrote:
Hmm, it sounds like a solution could be to separate timers that just
wake up a process from ones that do actual work and run them in separate
kernel threads.
On Tue, 2006-05-09 at 12:56 -0700, Kjetil S.
Matheussen wrote:
Well, the reason for stumble upon this problem is when I ported/tested
the general watchdog program "das_watchdog" from 2.4 to 2.6. For some
reason, it just didn't work. But after reading the instructions for
Florians rt_watchdog program, it became clear that the softirq
thread must have priority 99.
But lets say that you have program with two SCHED_FIFO threads. One
of them is a high priority (between 2 and 98) timer that is only
used to signal other threads. The other SCHED_FIFO thread has priority
1 and needs to run realtime for some reason, and it might not even be
a good reason. However, if that other thread have to do
some hard work, it stalls your high priority timer thread for a small
time, and your sleep-based timer suddenly becomes shaky.
Then theres a latency problem in the kernel. A sleeping high priorioty
SCHED_FIFO thread must be woken up in time even if another lower
priority SCHED_FIFO thread is buzy-looping. And currently, unless the
softirq timer has priority 99, that condition can not be fullfilled.
So, the softirq timer must run with priority 99.
I think it's unusual to want a high priority SCHED_FIFO thread to be
awakened by a timer. What are you trying to do?
8:57 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 2006-05-09 at 15:45 -0700, Kjetil S. Matheussen wrote:
Well as you have seen the softirq timer thread apparently does route
cache flushing and a bunch of other things we don't want it to do. I
was under the impression these were already done by separate threads.
I think we are in agreement. I have to double check how it works in -rt
- I've been running mainline lately which still has a single ksoftirqd
process.
Lee
Hmm, it sounds
like a solution could be to separate timers that just
wake up a process from ones that do actual work and run them in separate
kernel threads.
I don't understand why you want that. To me (which
knows about nothing about how the kernel works), the solution to
the problem is crystal clear: The softirq timer needs to have the highest
priority, and the only thing the sofirq timer threads does is to
make sure threads that should be woken up are put into some kernel
schedule queue somehow. I don't understand why the softirq timer threads
should cause any latency problems, what does it do except scheduling
waiting threads?
9:15 p.m.
New subject: [linux-audio-user] -rt IRQ handler priorities
On Tue, 9 May 2006, Lee Revell wrote:
On Tue, 2006-05-09 at 15:45 -0700, Kjetil S.
Matheussen wrote:
Well as you have seen the softirq timer thread apparently does route
cache flushing and a bunch of other things we don't want it to do. I
was under the impression these were already done by separate threads.
I think we are in agreement.
Good. :-)
Hmm, it
sounds like a solution could be to separate timers that just
wake up a process from ones that do actual work and run them in separate
kernel threads.
I don't understand why you want that. To me (which
knows about nothing about how the kernel works), the solution to
the problem is crystal clear: The softirq timer needs to have the highest
priority, and the only thing the sofirq timer threads does is to
make sure threads that should be woken up are put into some kernel
schedule queue somehow. I don't understand why the softirq timer threads
should cause any latency problems, what does it do except scheduling
waiting threads?
I have to double check how it works in -rt
- I've been running mainline lately which still has a single ksoftirqd
process.
I think the ksoftirqd processes have the same problem. At least for
das_watchdog, it must be set to 99 for the watchdog to work. There is
even code to check that it/they have priority 99.
6717
days inactive
6717
days old
linux-audio-user@lists.linuxaudio.org
7 comments
2 participants
participants (2)
-
Kjetil S. Matheussen -
Lee Revell