[linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the influence of linux1394 kernel drivers on scheduling latency
22 Jun
2006
22 Jun
'06
10:05 p.m.
Pieter Palmers wrote:
This weekend I've discovered a (serious) kernel
scheduling latency issue
with the current ieee1394 kernel drivers. Before I submit something
about this to lkml, I'd like some more tests. I've been able to
reproduce this on two different machines, so I suspect that this is a
more general problem.
I ran the tests quickly on my firewire development machine overnight. First
I started the latency monitor and left it run for about 10 minutes while I
did other things, including accessing a USB stick. I then started the
ieee1394 stress tester; almost immediately the monitor noted maximum
latencies of the order of 1 ms (compared to tens of microseconds when the
stress tester wasn't running). The output is blow.
./latency_histogram -p 80 -n 128 -t 1 -s 1
Histogram size: 128 bins, bin size: 1us, last bin: 128us
aquiring RT scheduling with priority 80 ...
getting cpu speed...
67547857.763 Hz (67.548 MHz)
Sleeping for 1us between timer reads...
Running, press CTRL-C to stop...
(+) New maximum: 62 cycles, 0.917868
(-) New minimum: 62 cycles, 0.917868
(-) New minimum: 58 cycles, 0.858650
(+) New maximum: 298 cycles, 4.411687
(+) New maximum: 43475 cycles, 643.617747
(+) New maximum: 48239 cycles, 714.145520
(+) New maximum: 52924 cycles, 783.503752
(+) New maximum: 131949 cycles, 1953.415021
(+) New maximum: 170136 cycles, 2518.747532
# of iterations: 243335
max. difference: 170136 cycles, 2518.747532us
min. difference: 58 cycles, 0.858650us
avg. difference: 66 cycles, 0.989601us
Histogram
---------
=< 1us: 33
1us : 243285
2us : 9
3us : 1
4us : 3
5us : 2
6us : 1
7us : 1
...
= 128us: 33
The "New maximum: 298 cycles" appeared very soon after the stress
tester was started, with the ones below it following soon after.
Despite what the log says, this was running a 2.0 GHz "Dothan" Centrino CPU.
Kernel was 2.6.16-rt25, distro was Slackware 10.2. Both the stress tester
and the monitor were run with RT privilege access. The firewire interface
used has a TI OHCI chipset.
I apologise that the run was particularly short and that therefore the
statistics aren't particularly good, but it does seem to confirm the
observations you made on your machine. The large latencies only occur
when the stress tester is running.
Regards
jonathan
22 Jun
22 Jun
10:11 p.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the influence of linux1394 kernel drivers on scheduling latency
On Fri, 2006-06-23 at 09:44 +0930, Jonathan Woithe wrote:
Despite what the log says, this was running a 2.0 GHz "Dothan"
Centrino CPU. Kernel was 2.6.16-rt25, distro was Slackware 10.2. Both
the stress tester and the monitor were run with RT privilege access.
The firewire interface used has a TI OHCI chipset.
I apologise that the run was particularly short and that therefore the
statistics aren't particularly good, but it does seem to confirm the
observations you made on your machine. The large latencies only occur
when the stress tester is running.
What if you run the latency tester at RT priority 99? Testing at 80 is
not particularly useful.
If anything else is running at 99, what happens if you lower those other
processes to 98?
What processes do you have running at RT priority 80 or above?
Does the Firewire stack do a lot of its work in timers?
Lee
10:41 p.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the
Hi Lee
On Fri, 2006-06-23 at 09:44 +0930, Jonathan Woithe
wrote:
Ok, I'll try at the higher number. I didn't have much time on this
overnight so I just ran it according to the instructions given in the README
file. I haven't had time to analyse the details.
Despite what the log says, this was running a 2.0
GHz "Dothan"
Centrino CPU. Kernel was 2.6.16-rt25, distro was Slackware 10.2. Both
the stress tester and the monitor were run with RT privilege access.
The firewire interface used has a TI OHCI chipset.
I apologise that the run was particularly short and that therefore the
statistics aren't particularly good, but it does seem to confirm the
observations you made on your machine. The large latencies only occur
when the stress tester is running.
What if you run the latency tester at RT priority 99? Testing at 80 is
not particularly useful. If anything else is running at 99, what happens if you
lower those other
processes to 98?
What processes do you have running at RT priority 80 or above?
At the time I ran the tests there were no other usermode processes running
at any RT priority. Only the latency analyser itself and the ieee stress
tester had elevated RT priorities. Also the system hasn't had priorities of
kernel things tuned on startup - the system is a bog-standard 2.6.16-rt25
running as booted.
The system as a whole was unloaded too; I had X running with fvwm2 and some
xterms running shells but that was about it.
Does the Firewire stack do a lot of its work in
timers?
Don't know - I'll have to check.
Regards
jonathan
10:55 p.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the
On Fri, 2006-06-23 at 10:19 +0930, Jonathan Woithe wrote:
At the time I ran the tests there were no other
usermode processes
running at any RT priority. Only the latency analyser itself and the
ieee stress tester had elevated RT priorities. Also the system hasn't
had priorities of kernel things tuned on startup - the system is a
bog-standard 2.6.16-rt25 running as booted.
In the -rt kernel IRQ handlers run as user processes with realtime
priority. You have to use ps to check.
Lee
11:07 p.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the
On Fri, 2006-06-23 at 10:19 +0930, Jonathan Woithe
wrote:
I see. I'll check. Note that in this case I haven't done any tuning, so
the priorities of the kernel IRQ handlers will be whatever an RT kernel sets
them to by default.
Regards
jonathan
At the time I ran the tests there were no other
usermode processes
running at any RT priority. Only the latency analyser itself and the
ieee stress tester had elevated RT priorities. Also the system hasn't
had priorities of kernel things tuned on startup - the system is a
bog-standard 2.6.16-rt25 running as booted.
In the -rt kernel IRQ handlers run as user processes with realtime
priority. You have to use ps to check. 
23 Jun
23 Jun
8:12 a.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the influence of linux1394 kernel drivers on scheduling latency
Lee Revell wrote:
On Fri, 2006-06-23 at 09:44 +0930, Jonathan Woithe
wrote:
Why not?
If the 1394 test user thread has a lower priority, and the ohci1394 irq
priority is also lower, there is no reason for the latency tester to be
preempted by them.
Despite what the log says, this was running a 2.0
GHz "Dothan"
Centrino CPU. Kernel was 2.6.16-rt25, distro was Slackware 10.2. Both
the stress tester and the monitor were run with RT privilege access.
The firewire interface used has a TI OHCI chipset.
I apologise that the run was particularly short and that therefore the
statistics aren't particularly good, but it does seem to confirm the
observations you made on your machine. The large latencies only occur
when the stress tester is running.
What if you run the latency tester at RT priority 99? Testing at 80 is
not particularly useful.
If anything else is running at 99, what happens if you
lower those other
processes to 98?
I'll have to recheck, but if I remember correcly I have done this
experiment. The only thing at 99 is the system timer. I tried giving it
a lower priority than the latency test thread, which didn't change anything.
Pieter
6:39 p.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the influence of linux1394 kernel drivers on scheduling latency
On Fri, 2006-06-23 at 12:12 +0200, Pieter Palmers wrote:
OK thanks, that answers my question.
Lee Revell wrote:
Because as you stated below the system timer runs at a higher priority
by default. I wanted to rule out preemption by the system timer thread.
On Fri, 2006-06-23 at 09:44 +0930, Jonathan Woithe
wrote:
Why not?
If the 1394 test user thread has a lower priority, and the ohci1394 irq
priority is also lower, there is no reason for the latency tester to be
preempted by them.
Despite what the log says, this was running a 2.0
GHz "Dothan"
Centrino CPU. Kernel was 2.6.16-rt25, distro was Slackware 10.2. Both
the stress tester and the monitor were run with RT privilege access.
The firewire interface used has a TI OHCI chipset.
I apologise that the run was particularly short and that therefore the
statistics aren't particularly good, but it does seem to confirm the
observations you made on your machine. The large latencies only occur
when the stress tester is running.
What if you run the latency tester at RT priority 99? Testing at 80 is
not particularly useful.
If anything
else is running at 99, what happens if you lower those other
processes to 98?
I'll have to recheck, but if I remember correcly I have done this
experiment. The only thing at 99 is the system timer. I tried giving it
a lower priority than the latency test thread, which didn't change anything.
Pieter
24 Jun
24 Jun
9:51 a.m.
New subject: [linux-audio-dev] Re: [Freebob-devel] [REQUEST] test the influence of linux1394 kernel drivers on scheduling latency
Lee Revell wrote:
On Fri, 2006-06-23 at 12:12 +0200, Pieter Palmers
wrote:
OK thanks, that answers my question.
I found one problem at this point, which is in the latency test
application itself. Apparently it doesn't acquire the RT priority
correctly. My mistake.
When trying with rtc_wakeup I get about 40us max. Which is more what I'd
expect.
I'm going to investigate this further anyway, because I'm not really
convinced yet. In one of my previous sessions, rtc_wakeup also showed
large latency values.
TBC
Greets,
Pieter
If
anything else is running at 99, what happens if you lower those other
processes to 98?
I'll have to recheck, but if I remember correcly I have done this
experiment. The only thing at 99 is the system timer. I tried giving it
a lower priority than the latency test thread, which didn't change anything.
6523
days inactive
6525
days old
linux-audio-dev@lists.linuxaudio.org
7 comments
3 participants
participants (3)
-
Jonathan Woithe -
Lee Revell -
Pieter Palmers