11 Jul
2004
11 Jul
'04
3:36 a.m.
Ingo Molnar <mingo(a)elte.hu> wrote:
I took a
look at latencies and indeed 2.6.7 is pretty bad - latencies up to 50
msec (!) can be easily triggered using common workloads, on fast 2GHz+
x86 system - even when using the fully preemptible kernel!
What were those workloads?
Certainly 2.6+preempt is not as good as 2.4+LL at this time, but 2.6 isn't
too bad either. Even under heavy filesystem load it's hard to exceed a 0.5
millisecond holdoff. There are still a few problem in the ext3 checkpoint
buffer handling, but those seem pretty hard to hit. I doubt if the `Jack'
testers were running `dbench 1000' during their testing.
All of which makes me suspect that the problems which the `Jack' testers
saw were not directly related to long periods of non-preemption in-kernel.
At least, not in core kernel/fs/mm code. There have been problem in the
past in places like i2c drivers, fbdev scrolling, etc.
What we need to do is to encourage audio testers to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
As for the patch, well, sprinkling rescheduling points everywhere is still
not the preferred approach. But adding more might_sleep() checks is a
sneaky way of making it more attractive ;)
Minor point: this:
cond_resched();
function_which_might_sleep();
is less efficient than
function_which_might_sleep();
cond_resched();
because if function_which_might_sleep() _does_ sleep, need_resched() will
likely be false when we hit cond_resched(), thus saving a context switch.
Unfortunately, might_sleep() calls tend to go at the entry to functions,
whereas cond_resched() calls should be neat the exit point, or inside loop
bodies.
11 Jul
11 Jul
7:40 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
Minor point: this:
cond_resched();
function_which_might_sleep();
is less efficient than
function_which_might_sleep();
cond_resched();
because if function_which_might_sleep() _does_ sleep, need_resched()
will likely be false when we hit cond_resched(), thus saving a context
switch. Unfortunately, might_sleep() calls tend to go at the entry to
functions, whereas cond_resched() calls should be neat the exit point,
or inside loop bodies.
agreed, but the argument goes both ways. Whether entry or exit
scheduling is better very much depends on the function.
E.g. for user copy type of stuff we often want to do the reschedule
_first_, to not pollute the cache with hot (dirty) cachelines that
likely get thrown away - and which have to be brought back again later
on.
For IO type of functions that will sleep we most likely want to preempt
at the exit of the function.
but we'd like to profile the typical preemption points (hence the
profile=sched profiling change) to determine which are the hottest
functions. For those handful of functions we can do __might_sleep() at
the front of the function and a cond_resched() at the back. For all the
other 200 might_sleep() points it doesnt matter much.
i've also added the nr_preempt counter so that we can see the proportion
of forced preemption vs. intentional reschedules for various workloads.
Ingo
7:56 a.m.
Ingo Molnar <mingo(a)elte.hu> wrote:
For all the
other 200 might_sleep() points it doesnt matter much.
Sorry, but an additional 100 might_sleep()s is surely excessive for
debugging purposes, and unneeded for latency purposes: all these sites are
preemptible anyway.
Let me repeat that I am unconvinced as to the diagnosis of the current
audio problems - more analysis might prove me wrong of course.
And I'm unconvinced that we need to do anything apart from identifying and
fixing the remaining spinlocks which are holding off preemption for too
long.
IOW, I am questioning the very need for a "voluntary preemption" feature
at all when "involuntary preemption" works perfectly well.
8:04 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
Ingo Molnar <mingo(a)elte.hu> wrote:
nono, i mean the existing ones. (it's 116 not 200) There's no plan to
add another 100, you've seen all the ones we found to be necessary for
this.
Ingo
For all the
other 200 might_sleep() points it doesnt matter much.
Sorry, but an additional 100 might_sleep()s is surely excessive for
debugging purposes, and unneeded for latency purposes: all these sites
are preemptible anyway.
8:10 a.m.
Ingo Molnar <mingo(a)elte.hu> wrote:
* Andrew Morton <akpm(a)osdl.org> wrote:
OK, but most of the new ones are unneeded with CONFIG_PREEMPT=y. I'm still
failing to see why a non-preempt, voluntary preemption kernel even needs to
try to be competitive with a preemptible kernel?
Ingo Molnar <mingo(a)elte.hu> wrote:
nono, i mean the existing ones. (it's 116 not 200) There's no plan to
add another 100, you've seen all the ones we found to be necessary for
this.
For all the
other 200 might_sleep() points it doesnt matter much.
Sorry, but an additional 100 might_sleep()s is surely excessive for
debugging purposes, and unneeded for latency purposes: all these sites
are preemptible anyway.
8:42 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
OK, but most of the new ones are unneeded with
CONFIG_PREEMPT=y. I'm
still failing to see why a non-preempt, voluntary preemption kernel
even needs to try to be competitive with a preemptible kernel?
the reason is difference in overhead (codesize, speed) and risks (driver
robustness). We do not want to enable preempt for Fedora yet because it
breaks just too much stuff and is too heavy. So we looked for a solution
that might work for a generic distro.
here are the code size differences. With a typical .config (debugging
options disabled), the 2.6.7-mm7(+voluntary-preempt) UP x86 kernel gets
the following .text sizes:
orig: 1776911 bytes
preempt: 1855519 bytes (+4.4%)
voluntary: 1783407 bytes (+0.3%)
so if voluntary-preempt can get close to real preempt's numbers for
practical stuff then we get most of the benefits while excluding some of
the nastiest risks and disadvantages.
(Long-term i'd like to see preempt be used unconditionally - at which
point the 10-line CONFIG_VOLUNTARY_PREEMPT Kconfig and kernel.h change
could go away.)
Ingo
8:54 a.m.
Ingo Molnar <mingo(a)elte.hu> wrote:
* Andrew Morton <akpm(a)osdl.org> wrote:
I don't recall any testing results which showed a significant performance
difference from CONFIG_PREEMPT.
OK, but most of the new ones are unneeded with
CONFIG_PREEMPT=y. I'm
still failing to see why a non-preempt, voluntary preemption kernel
even needs to try to be competitive with a preemptible kernel?
the reason is difference in overhead (codesize, speed) and risks (driver
robustness). We do not want to enable preempt for Fedora yet
because it
breaks just too much stuff
What stuff?
(Long-term i'd like to see preempt be used
unconditionally - at which
point the 10-line CONFIG_VOLUNTARY_PREEMPT Kconfig and kernel.h change
could go away.)
We'll never get there if people don't at least report the broken
"stuff",
let alone fix it. And "stuff" is already broken on SMP, yes?
Your voluntary preempt patch will need to borrow preempt_spin_lock() and
preempt_write_lock() btw - otherwise it won't improve worst-case latencies
on SMP at all.
9:10 a.m.
On Sun, Jul 11, 2004 at 03:42:58AM -0700, Andrew Morton wrote:
just look over all the "fix preempt" stuff that got added to the kernel in
the last 6 months. Sometimes subtle sometimes less so. From a distribution
POV I don't want a potential slew of basically impossible to reproduce
problems, especially this young in 2.6, there are plenty of other problems
already (and before you ask "which", just look at how many bugs got fixed in
the last X weeks for any value of X, and look at say acpi issues).
Yes I understand this puts you into a bit of a bad position, several distros
not enabling preempt means that it gets less testing than it should.
However.. there's only so much issues distros can take and with 2.6 still
quite fresh...
That's the classic preempt "myth"; it's true if you ignore per cpu stuff
and
some other subtle issues ;) And even then, yes a lot of our drivers are not
quite SMP safe. Take ISDN or any of the other declared SMP-broken drivers.
Not to speak of the ones that aren't declared as such yet still are.
Regardless of Hyperthreading, smp is still quite rare while crappy
hardware/drivers are not.
Do the BROKEN_ON_SMP tests get triggered in Kconfig for PREEMPT ? It probably
should.
Greetings,
Arjan van de Ven
We do not want
to enable preempt for Fedora yet because it
breaks just too much stuff
What stuff? (Long-term
i'd like to see preempt be used unconditionally - at which
point the 10-line CONFIG_VOLUNTARY_PREEMPT Kconfig and kernel.h change
could go away.)
And "stuff" is already broken on SMP, yes?
9:24 a.m.
Arjan van de Ven <arjanv(a)redhat.com> wrote:
That's the classic preempt "myth"; it's true if you ignore per cpu
stuff and
some other subtle issues ;)
?
Sticking a WARN_ON(!in_atomic()) if CONFIG_PREEMPT into smp_processor_id()
should catch any missed fixes.
On Sun, Jul 11, 2004 at 03:42:58AM -0700, Andrew Morton wrote:
just look over all the "fix preempt" stuff that got added to the kernel in
the last 6 months. Sometimes subtle sometimes less so. From a distribution
POV I don't want a potential slew of basically impossible to reproduce
problems, especially this young in 2.6, there are plenty of other problems
already (and before you ask "which", just look at how many bugs got fixed in
the last X weeks for any value of X, and look at say acpi issues).
Yes I understand this puts you into a bit of a bad position, several distros
not enabling preempt means that it gets less testing than it should.
However.. there's only so much issues distros can take and with 2.6 still
quite fresh...
IOW: "we haven't found any such stuff". Sounds fuddy to me.
We do not
want to enable preempt for Fedora yet because it
breaks just too much stuff
What stuff? (Long-term i'd like to see preempt be used
unconditionally - at which
point the 10-line CONFIG_VOLUNTARY_PREEMPT Kconfig and kernel.h change
could go away.)
And "stuff" is already broken on SMP, yes? And even then, yes a lot of our drivers are not
quite SMP safe. Take ISDN or any of the other declared SMP-broken drivers.
Not to speak of the ones that aren't declared as such yet still are.
Regardless of Hyperthreading, smp is still quite rare while crappy
hardware/drivers are not.
Oh I can buy the make-the-bugs-less-probable practical argument, but
sheesh. If you insist on going this way we can stick the patch in after
2.7 has forked. I spose. The patch will actually slow the rate of
improvement of the kernel :(
10:12 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
Oh I can buy the make-the-bugs-less-probable practical
argument, but
sheesh. If you insist on going this way we can stick the patch in
after 2.7 has forked. I spose. The patch will actually slow the rate
of improvement of the kernel :(
it has already improved the kernel by already detecting at least 2 bugs
that were not found via CONFIG_PREEMPT or other means (the cfq-iosched
one and the proc_pid_flush() one) - less then 1 week after i wrote the
patch.
It also already resulted in a number of independent improvements: e.g.
the lock-break changes it includes, and all the work on the debug
infrastructure.
So i fail to see how it would slow the rate of improvement of the
kernel. It's simply the middle step towards CONFIG_PREEMPT - a feature
that several major distribution have not enabled so far. It's a more
opt-in approach instead of CONFIG_PREEMPT's opt-out approach, that's
all.
(that having said it's not at all clear that voluntary-preempt would get
into Fedora. If it causes stability problems it will be zapped. But at
least the problems it triggers we could trigger _even on the current UP
kernel_, and that's the difference to preempt - CONFIG_PREEMPT triggers
_new_ problems on UP.)
also, frankly, at minimum the latency side of CONFIG_PREEMPT has been
underdeveloped lately. Nobody seemed to care about real latencies that
happen. Just do a 'du /; du /; sync' on a box with enough RAM and listen
to audio. Or trigger some swapping. (these particular latencies are
fixed in the voluntary-preempt patchset.)
Ingo
4:17 p.m.
On Sun, 11 Jul 2004, Andrew Morton wrote:
Arjan van de Ven <arjanv(a)redhat.com> wrote:
For example the recent race that corrupted file content on ext3 and
reiserfs when fsync and write were called simultaneously ... it was
possible on SMP too, but with tiny probability --- CONFIG_PREEMPT
triggered it wide open.
Mikulas
On Sun, Jul 11, 2004 at 03:42:58AM -0700, Andrew Morton wrote:
just look over all the "fix preempt" stuff that got added to the kernel in
the last 6 months. Sometimes subtle sometimes less so. From a distribution
POV I don't want a potential slew of basically impossible to reproduce
problems, especially this young in 2.6, there are plenty of other problems
already (and before you ask "which", just look at how many bugs got fixed in
the last X weeks for any value of X, and look at say acpi issues).
Yes I understand this puts you into a bit of a bad position, several distros
not enabling preempt means that it gets less testing than it should.
However.. there's only so much issues distros can take and with 2.6 still
quite fresh...
IOW: "we haven't found any such stuff". Sounds fuddy to me. We do not
want to enable preempt for Fedora yet because it
breaks just too much stuff
What stuff? 
12 Jul
12 Jul
5:15 a.m.
On Sun, 11 Jul 2004, Arjan van de Ven wrote:
On Sun, Jul 11, 2004 at 03:42:58AM -0700, Andrew
Morton wrote:
> > We do not want to enable preempt for Fedora yet because it
> > breaks just too much stuff
>
> What stuff?
Looks like some of the voluntary preemption changes might need some
eyeballing too. This appears to be a use after free, probably since we
unlocked j_{list,state}_lock.
Unable to handle kernel paging request at virtual address d90ccfa0
printing eip:
c01e667f
*pde = 00066067
Oops: 0000 [#1]
PREEMPT SMP DEBUG_PAGEALLOC
Modules linked in:
CPU: 1
EIP: 0060:[<c01e667f>] Not tainted
EFLAGS: 00010202 (2.6.7-bk20-ck5)
EIP is at __journal_clean_checkpoint_list+0x3f/0x1d0
eax: 00000001 ebx: 00000000 ecx: df34bdf8 edx: d90ccf78
esi: df34be0c edi: df34bedc ebp: de893dac esp: de893d88
ds: 007b es: 007b ss: 0068
Process kjournald (pid: 36, threadinfo=de892000 task=deea8a50)
Stack: de892000 d23fa7e4 00000172 d91a9f78 c7da0f78 d90ccf78 de892000 df34bdf8
df34be7c de893f50 c01e3956 df34bdf8 c014cc71 dffb1200 c4cccf60 00000046
de892000 df34bedc df1d1e18 c4cccf60 d8b0efb0 df34be0c 00000000 00000000
Call Trace:
[<c0107775>] show_stack+0x75/0x90
[<c01078d5>] show_registers+0x125/0x180
[<c0107a67>] die+0xa7/0x170
[<c011ad08>] do_page_fault+0x1e8/0x57d
[<c01073cd>] error_code+0x2d/0x40
[<c01e3956>] journal_commit_transaction+0x316/0x1aa0
[<c01e7994>] kjournald+0x114/0x3b0
[<c01042a5>] kernel_thread_helper+0x5/0x10
Code: 8b 72 28 85 f6 74 58 8b 4e 2c b8 00 e0 ff ff 89 4d e0 89 f7
<6>note: kjournald[36] exited with preempt_count 2
(gdb) list *0xc01e667f
0xc01e667f is in __journal_clean_checkpoint_list (fs/jbd/checkpoint.c:485).
480 struct journal_head *jh;
481
482 transaction = next_transaction;
483 next_transaction = transaction->t_cpnext;
484 retry:
485 jh = transaction->t_checkpoint_list;
486 if (jh) {
487 struct journal_head *last_jh = jh->b_cpprev;
488 struct journal_head *next_jh = jh;
489
19 Jul
19 Jul
8:47 a.m.
* Zwane Mwaikambo <zwane(a)linuxpower.ca> wrote:
Looks like some of the voluntary preemption changes
might need some
eyeballing too. This appears to be a use after free, probably since we
unlocked j_{list,state}_lock.
thx - this is the new __journal_clean_checkpoint_list() chunk:
--- linux/fs/jbd/checkpoint.c.orig
+++ linux/fs/jbd/checkpoint.c
@@ -465,6 +470,7 @@ int __journal_clean_checkpoint_list(jour
transaction_t *transaction, *last_transaction, *next_transaction;
int ret = 0;
+retry:
transaction = journal->j_checkpoint_transactions;
if (transaction == 0)
goto out;
@@ -487,6 +493,14 @@ int __journal_clean_checkpoint_list(jour
/* Use trylock because of the ranknig */
if (jbd_trylock_bh_state(jh2bh(jh)))
ret += __try_to_free_cp_buf(jh);
+ if (voluntary_need_resched()) {
+ spin_unlock(&journal->j_list_lock);
+ spin_unlock(&journal->j_state_lock);
+ voluntary_resched();
+ spin_lock(&journal->j_state_lock);
+ spin_lock(&journal->j_list_lock);
+ goto retry;
+ }
} while (jh != last_jh);
}
} while (transaction != last_transaction);
11 Jul
11 Jul
9:21 p.m.
On Sun, 2004-07-11 at 12:30 +0200, Ingo Molnar wrote:
the reason is difference in overhead (codesize, speed)
and risks (driver
robustness). We do not want to enable preempt for Fedora yet because it
breaks just too much stuff and is too heavy. So we looked for a solution
that might work for a generic distro.
I think we should work toward being able to enable kernel preemption in
Fedora, then, instead of other tangential solutions.
And I disagree with the overhead argument. I have seen no specific
arguments that show a significant overhead. Heck, when people tried to
show that kernel preemption hurt throughput, we saw tests that showed
improved throughput (probably due to better utilization of I/O).
But stability is a subjective argument (and I agree we need more driver
love, at least for obscure drivers) wrt kernel preemption. So I would
say we should concentrate on working on the stability[1] so we could
just enable kernel preemption unconditionally and not designing new
solutions.
Best,
Robert Love
[1] What better way than enabling CONFIG_PREEMPT for Fedora? Enable it
for Fedora, and do not enable it for Red Hat Enterprise until you are
confidant. ;-)
12 Jul
12 Jul
11:34 a.m.
Andrew> Ingo Molnar <mingo(a)elte.hu> wrote:
> I took a look at latencies and indeed 2.6.7 is
pretty bad -
> latencies up to 50 msec (!) can be easily triggered using common
> workloads, on fast 2GHz+ x86 system - even when using the fully
> preemptible kernel!
Andrew> What were those workloads?
Andrew> Certainly 2.6+preempt is not as good as 2.4+LL at this time,
Andrew> but 2.6 isn't too bad either. Even under heavy filesystem
Andrew> load it's hard to exceed a 0.5 millisecond holdoff. There are
Andrew> still a few problem in the ext3 checkpoint buffer handling,
Andrew> but those seem pretty hard to hit. I doubt if the `Jack'
Andrew> testers were running `dbench 1000' during their testing.
I'd like to chime in here with a data point, though I'm not sure how
usefull it really is. I've got a Dual CPU, 550mhz, 768mb 440BX
system. Running 2.6.7 plain. I was playing XMMS and some music, and
trying to do the following from my mirrored data disks, to a new USB
external storage case, with the exact same drive as the pair of
internal systems.
cd /home
tar cf - . | (cd /mnt/portable; tar xpf -)
And the system load jumped to around 5 or 7. Part of it was because I
had turned on usb-storage debugging, so syslog was hammering the
disks. Note, root, /usr, and such are on a seperate SCSI disk from
/home and /local. I had xmms, some xterms and galeon up and running.
Occassionally, xmms would stutter. Then the entire system locked up
with xmms stuck in a loop playing the same sound over and over again.
No response on the USB bus, SysRq didn't work. Had to hit the main
reset button to get it back again.
All filesystems are ext3 on here as well. More details as I get this
tested out more, and I'll try to fill out a report on the usb-devel
list as well.
So in summary, there are some problems still in 2.6.7, though they are
more related to:
1. The delay in starting writeout to disk when streaming large (I
know, a subjective number) transfers. The above tar commands didn't
start kicking data to the USB device for quite a while. I know
there's overheard of reading the directory structure first, so that
could be it. But it just seems to wait a bit before actually writing
data out.
John
9:06 p.m.
On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
What we need to do is to encourage audio testers to
use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
OK, here goes. The following traces result from running JACK overnight
like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with
a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch.
voluntary_preempt and kernel_preemption are both on.
jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
--playback --period 32 --nperiods 2
These settings require less than 666 microseconds scheduler latency.
The average performance is quite good - 5-20 *microseconds*!
load = 0.3754 max usecs: 5.000, spare = 661.000
load = 1.1637 max usecs: 13.000, spare = 653.000
load = 3.0593 max usecs: 33.000, spare = 633.000
load = 1.9050 max usecs: 5.000, spare = 661.000
load = 1.4780 max usecs: 7.000, spare = 659.000
load = 1.2645 max usecs: 7.000, spare = 659.000
load = 1.0076 max usecs: 5.000, spare = 661.000
load = 1.1044 max usecs: 8.000, spare = 658.000
load = 0.9276 max usecs: 5.000, spare = 661.000
load = 1.5148 max usecs: 14.000, spare = 652.000
load = 1.1328 max usecs: 5.000, spare = 661.000
load = 1.0168 max usecs: 6.000, spare = 660.000
However there are periodic XRUNS. I noticed that many of these seem
APM-related, so here are the relevant settings:
# Power management options (ACPI, APM)
# APM (Advanced Power Management) BIOS Support
CONFIG_APM=y
# CONFIG_APM_IGNORE_USER_SUSPEND is not set
# CONFIG_APM_DO_ENABLE is not set
CONFIG_APM_CPU_IDLE=y
# CONFIG_APM_DISPLAY_BLANK is not set
CONFIG_APM_RTC_IS_GMT=y
# CONFIG_APM_ALLOW_INTS is not set
# CONFIG_APM_REAL_MODE_POWER_OFF is not set
I did not intend to activate these settings when I compiled the kernel,
but as this is certainly not an idle system, it seems like there may be
a bug lurking.
The log excerpts below are annotated. Sorry if it is too long.
This section of the log is from running JACK overnight:
Jul 12 06:56:07 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 06:56:07 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 06:56:07 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 06:56:07 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 06:56:07 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 06:56:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 06:56:07 mindpipe kernel: [preempt_schedule+16/80] preempt_schedule+0x10/0x50
Jul 12 06:56:07 mindpipe kernel: [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 06:56:07 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 06:56:07 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 06:56:07 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 06:56:07 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 06:56:07 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 06:56:07 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 06:56:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 06:56:07 mindpipe kernel:
Jul 12 07:46:07 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 07:46:07 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 07:46:07 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 07:46:07 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 07:46:07 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 07:46:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 07:46:07 mindpipe kernel: [preempt_schedule+1/80] preempt_schedule+0x1/0x50
Jul 12 07:46:07 mindpipe kernel: [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 07:46:07 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 07:46:07 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 07:46:07 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 07:46:07 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 07:46:07 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 07:46:07 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 07:46:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 07:46:07 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 07:46:07 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 07:46:07 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 07:46:07 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 07:46:07 mindpipe kernel:
Jul 12 08:06:07 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:06:07 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:06:07 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:06:07 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:06:07 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:06:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:06:07 mindpipe kernel: [local_bh_enable+46/112] local_bh_enable+0x2e/0x70
Jul 12 08:06:07 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 08:06:07 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:06:07 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:06:07 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 08:06:07 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:06:07 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:06:07 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:06:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:06:07 mindpipe kernel:
Jul 12 08:26:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:26:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:26:06 mindpipe kernel: [ide_build_sglist+50/160] ide_build_sglist+0x32/0xa0
Jul 12 08:26:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:26:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:26:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:26:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:26:06 mindpipe kernel: [del_timer+110/144] del_timer+0x6e/0x90
Jul 12 08:26:06 mindpipe kernel: [schedule_timeout+94/176] schedule_timeout+0x5e/0xb0
Jul 12 08:26:06 mindpipe kernel: [process_timeout+0/16] process_timeout+0x0/0x10
Jul 12 08:26:06 mindpipe kernel: [do_poll+161/192] do_poll+0xa1/0xc0
Jul 12 08:26:06 mindpipe kernel: [sys_poll+330/560] sys_poll+0x14a/0x230
Jul 12 08:26:06 mindpipe kernel: [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 08:26:06 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 08:26:06 mindpipe kernel:
Jul 12 08:36:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:36:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:36:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:36:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:36:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:36:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:36:06 mindpipe kernel: [rt_run_flush+66/144] rt_run_flush+0x42/0x90
Jul 12 08:36:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:36:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:36:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 08:36:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:36:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:36:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:36:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:36:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:36:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:36:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:36:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:36:06 mindpipe kernel:
Jul 12 08:46:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:46:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:46:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:46:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:46:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:46:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:46:06 mindpipe kernel: [rt_run_flush+123/144] rt_run_flush+0x7b/0x90
Jul 12 08:46:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:46:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:46:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 08:46:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:46:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:46:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:46:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:46:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:46:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:46:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:46:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:46:06 mindpipe kernel:
Jul 12 08:56:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 08:56:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 08:56:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 08:56:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 08:56:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 08:56:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:56:06 mindpipe kernel: [rt_run_flush+123/144] rt_run_flush+0x7b/0x90
Jul 12 08:56:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 08:56:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 08:56:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 08:56:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 08:56:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 08:56:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 08:56:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 08:56:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 08:56:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 08:56:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 08:56:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 08:56:06 mindpipe kernel:
Jul 12 09:06:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:06:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:06:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:06:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:06:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:06:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:06:06 mindpipe kernel: [preempt_schedule+1/80] preempt_schedule+0x1/0x50
Jul 12 09:06:06 mindpipe kernel: [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 09:06:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:06:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:06:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 09:06:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:06:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:06:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:06:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:06:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:06:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:06:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:06:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:06:06 mindpipe kernel:
Jul 12 09:36:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:36:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:36:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:36:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:36:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:36:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:36:06 mindpipe kernel: [local_bh_enable+23/112] local_bh_enable+0x17/0x70
Jul 12 09:36:06 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 09:36:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:36:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:36:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 09:36:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:36:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:36:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:36:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:36:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:36:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:36:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:36:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:36:06 mindpipe kernel:
Jul 12 09:46:06 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 09:46:06 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 09:46:06 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 09:46:06 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 09:46:06 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 09:46:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:46:06 mindpipe kernel: [preempt_schedule+23/80] preempt_schedule+0x17/0x50
Jul 12 09:46:06 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 09:46:06 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 09:46:06 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 09:46:06 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 09:46:06 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 09:46:06 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 09:46:06 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 09:46:06 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 09:46:06 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 09:46:06 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 09:46:06 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 09:46:06 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 09:46:06 mindpipe kernel:
Jul 12 10:16:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:16:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:16:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:16:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:16:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:16:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:16:05 mindpipe kernel: [preempt_schedule+73/80] preempt_schedule+0x49/0x50
Jul 12 10:16:05 mindpipe kernel: [rt_run_flush+135/144] rt_run_flush+0x87/0x90
Jul 12 10:16:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:16:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:16:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 10:16:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:16:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:16:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:16:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:16:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:16:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:16:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:16:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:16:05 mindpipe kernel:
Jul 12 10:46:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:46:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:46:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:46:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:46:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:46:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:46:05 mindpipe kernel: [rt_run_flush+94/144] rt_run_flush+0x5e/0x90
Jul 12 10:46:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:46:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:46:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 10:46:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:46:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:46:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:46:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:46:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:46:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:46:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:46:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:46:05 mindpipe kernel:
Jul 12 10:56:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 10:56:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 10:56:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 10:56:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 10:56:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 10:56:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:56:05 mindpipe kernel: [local_bh_enable+8/112] local_bh_enable+0x8/0x70
Jul 12 10:56:05 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 10:56:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 10:56:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 10:56:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 10:56:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 10:56:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 10:56:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 10:56:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 10:56:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 10:56:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 10:56:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 10:56:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 10:56:05 mindpipe kernel:
Jul 12 11:06:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:06:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:06:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:06:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:06:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:06:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:06:05 mindpipe kernel: [preempt_schedule+9/80] preempt_schedule+0x9/0x50
Jul 12 11:06:05 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 11:06:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:06:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:06:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 11:06:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:06:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:06:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:06:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:06:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:06:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:06:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:06:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 11:06:05 mindpipe kernel:
Jul 12 11:36:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:36:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:36:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:36:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:36:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:36:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:36:05 mindpipe kernel: [local_bh_enable+0/112] local_bh_enable+0x0/0x70
Jul 12 11:36:05 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 12 11:36:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:36:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:36:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 11:36:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:36:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:36:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:36:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:36:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:36:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:36:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:36:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 12 11:36:05 mindpipe kernel:
Jul 12 11:46:05 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 11:46:05 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 11:46:05 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 11:46:05 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 11:46:05 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 11:46:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:46:05 mindpipe kernel: [rt_run_flush+66/144] rt_run_flush+0x42/0x90
Jul 12 11:46:05 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 12 11:46:05 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 12 11:46:05 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 12 11:46:05 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 12 11:46:05 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 12 11:46:05 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 12 11:46:05 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 11:46:05 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 12 11:46:05 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 12 11:46:05 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 12 11:46:05 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
That continues until I sat down at the machine and switched consoles. I am aware that
this is a known latency 'hot spot' and not trivial to fix. Here are the traces
anyway:
Jul 12 17:33:42 mindpipe kernel:
Jul 12 17:33:42 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p Jul 12
17:33:42 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:33:42 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:33:42 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:33:42 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:33:42 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:33:42 mindpipe kernel: [release_console_sem+129/208]
release_console_sem+0x81/0xd0
Jul 12 17:33:42 mindpipe kernel: [do_con_write+657/1888] do_con_write+0x291/0x760
Jul 12 17:33:42 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
Jul 12 17:33:42 mindpipe kernel: [opost+162/464] opost+0xa2/0x1d0
Jul 12 17:33:42 mindpipe kernel: [write_chan+437/544] write_chan+0x1b5/0x220
Jul 12 17:33:42 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 12 17:33:42 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 12 17:33:42 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 12 17:33:42 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 12 17:33:42 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:33:42 mindpipe kernel:
Jul 12 17:33:42 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:33:42 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:33:42 mindpipe kernel: [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 17:33:42 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:33:42 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:33:42 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:33:42 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:33:42 mindpipe kernel: [lf+1/112] lf+0x1/0x70
Jul 12 17:33:42 mindpipe kernel: [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 12 17:33:42 mindpipe kernel: [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 12 17:33:42 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
Jul 12 17:33:42 mindpipe kernel: [opost+162/464] opost+0xa2/0x1d0
Jul 12 17:33:42 mindpipe kernel: [write_chan+437/544] write_chan+0x1b5/0x220
Jul 12 17:33:42 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 12 17:33:42 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 12 17:33:42 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 12 17:33:42 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 12 17:33:42 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 12 17:33:42 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:33:42 mindpipe kernel:
It looks like reiserfs has a hot spot as well:
Jul 12 17:55:11 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel: [do_pollfd+125/144] do_pollfd+0x7d/0x90
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel: [free_hot_cold_page+21/288]
free_hot_cold_page+0x15/0x120
Jul 12 17:55:11 mindpipe kernel: [poll_freewait+68/80] poll_freewait+0x44/0x50
Jul 12 17:55:11 mindpipe kernel: [sys_poll+489/560] sys_poll+0x1e9/0x230
Jul 12 17:55:11 mindpipe kernel: [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 17:55:11 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel: [can_dirty+99/192] can_dirty+0x63/0xc0
Jul 12 17:55:11 mindpipe kernel: [dirty_one_transaction+74/128]
dirty_one_transaction+0x4a/0x80
Jul 12 17:55:11 mindpipe kernel: [flush_commit_list+519/944]
flush_commit_list+0x207/0x3b0
Jul 12 17:55:11 mindpipe kernel: [do_journal_end+2081/2848] do_journal_end+0x821/0xb20
Jul 12 17:55:11 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:11 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:11 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:11 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:11 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:11 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:11 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:11 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:11 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:11 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:11 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:11 mindpipe kernel: [finish_task_switch+34/112]
finish_task_switch+0x22/0x70
Jul 12 17:55:11 mindpipe kernel: [schedule+731/1424] schedule+0x2db/0x590
Jul 12 17:55:11 mindpipe kernel: [__mod_timer+206/384] __mod_timer+0xce/0x180
Jul 12 17:55:11 mindpipe kernel: [schedule_timeout+88/176] schedule_timeout+0x58/0xb0
Jul 12 17:55:11 mindpipe kernel: [process_timeout+0/16] process_timeout+0x0/0x10
Jul 12 17:55:11 mindpipe kernel: [do_poll+161/192] do_poll+0xa1/0xc0
Jul 12 17:55:11 mindpipe kernel: [sys_poll+330/560] sys_poll+0x14a/0x230
Jul 12 17:55:11 mindpipe kernel: [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 12 17:55:11 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:11 mindpipe kernel:
Jul 12 17:55:17 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:17 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:17 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:17 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:17 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:17 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:17 mindpipe kernel: [remove_journal_hash+72/208]
remove_journal_hash+0x48/0xd0
Jul 12 17:55:17 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:55:17 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:55:17 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:55:17 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:55:17 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:55:17 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:55:17 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:55:17 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:17 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:17 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:17 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:17 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:17 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:17 mindpipe kernel:
Jul 12 17:55:17 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:55:17 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:55:17 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:55:17 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:55:17 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:55:17 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:55:17 mindpipe kernel: [remove_journal_hash+67/208]
remove_journal_hash+0x43/0xd0
Jul 12 17:55:17 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:55:17 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:55:17 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:55:17 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:55:17 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:55:17 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:55:17 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:55:17 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:55:17 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:55:17 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:55:17 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:55:17 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:55:17 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:55:17 mindpipe kernel:
A few minutes later:
Jul 12 17:56:54 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:56:54 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:56:54 mindpipe kernel: [i8042_timer_func+0/16] i8042_timer_func+0x0/0x10
Jul 12 17:56:54 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:56:54 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:56:54 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:56:54 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:56:54 mindpipe kernel: [can_dirty+144/192] can_dirty+0x90/0xc0
Jul 12 17:56:54 mindpipe kernel: [dirty_one_transaction+74/128]
dirty_one_transaction+0x4a/0x80
Jul 12 17:56:54 mindpipe kernel: [flush_commit_list+519/944]
flush_commit_list+0x207/0x3b0
Jul 12 17:56:54 mindpipe kernel: [do_journal_end+2081/2848] do_journal_end+0x821/0xb20
Jul 12 17:56:54 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:56:54 mindpipe kernel: [reiserfs_sync_fs+61/128] reiserfs_sync_fs+0x3d/0x80
Jul 12 17:56:54 mindpipe kernel: [sync_supers+174/192] sync_supers+0xae/0xc0
Jul 12 17:56:54 mindpipe kernel: [wb_kupdate+47/256] wb_kupdate+0x2f/0x100
Jul 12 17:56:54 mindpipe kernel: [deactivate_task+29/64] deactivate_task+0x1d/0x40
Jul 12 17:56:54 mindpipe kernel: [__pdflush+200/496] __pdflush+0xc8/0x1f0
Jul 12 17:56:54 mindpipe kernel: [pdflush+0/48] pdflush+0x0/0x30
Jul 12 17:56:54 mindpipe kernel: [pdflush+30/48] pdflush+0x1e/0x30
Jul 12 17:56:54 mindpipe kernel: [wb_kupdate+0/256] wb_kupdate+0x0/0x100
Jul 12 17:56:54 mindpipe kernel: [kthread+163/176] kthread+0xa3/0xb0
Jul 12 17:56:54 mindpipe kernel: [kthread+0/176] kthread+0x0/0xb0
Jul 12 17:56:54 mindpipe kernel: [kernel_thread_helper+5/16]
kernel_thread_helper+0x5/0x10
Jul 12 17:56:54 mindpipe kernel:
Another one:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+67/208]
remove_journal_hash+0x43/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+72/208]
remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+99/208]
remove_journal_hash+0x63/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+72/208]
remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+72/208]
remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+67/208]
remove_journal_hash+0x43/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+108/208]
remove_journal_hash+0x6c/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_used_journal_lists+147/160]
flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb Jul 12 17:57:22
mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 17:57:22 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [remove_journal_hash+72/208]
remove_journal_hash+0x48/0xd0
Jul 12 17:57:22 mindpipe kernel: [remove_all_from_journal_list+78/160]
remove_all_from_journal_list+0x4e/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+496/1344]
flush_journal_list+0x1f0/0x540
Jul 12 17:57:22 mindpipe kernel: [flush_older_journal_lists+48/64]
flush_older_journal_lists+0x30/0x40
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+795/1344]
flush_journal_list+0x31b/0x540 Jul 12 17:57:22 mindpipe kernel:
[flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 17:57:22 mindpipe kernel:
Jul 12 17:57:22 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 17:57:22 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1] Jul 12 17:57:22 mindpipe kernel:
[handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 17:57:22 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 17:57:22 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 17:57:22 mindpipe kernel: [generic_unplug_device+32/96]
generic_unplug_device+0x20/0x60
Jul 12 17:57:22 mindpipe kernel: [blk_backing_dev_unplug+35/48]
blk_backing_dev_unplug+0x23/0x30
Jul 12 17:57:22 mindpipe kernel: [sync_buffer+53/64] sync_buffer+0x35/0x40
Jul 12 17:57:22 mindpipe kernel: [__wait_on_buffer+150/160] __wait_on_buffer+0x96/0xa0
Jul 12 17:57:22 mindpipe kernel: [bh_wake_function+0/64] bh_wake_function+0x0/0x40
Jul 12 17:57:22 mindpipe kernel: [bh_wake_function+0/64] bh_wake_function+0x0/0x40
Jul 12 17:57:22 mindpipe kernel: [_update_journal_header_block+177/240]
_update_journal_header_block+0xb1/0xf0
Jul 12 17:57:22 mindpipe kernel: [update_journal_header_block+21/48]
update_journal_header_block+0x15/0x30
Jul 12 17:57:22 mindpipe kernel: [flush_journal_list+835/1344]
flush_journal_list+0x343/0x540 Jul 12 17:57:22 mindpipe kernel:
[flush_used_journal_lists+147/160] flush_used_journal_lists+0x93/0xa0
Jul 12 17:57:22 mindpipe kernel: [flush_old_journal_lists+69/112]
flush_old_journal_lists+0x45/0x70
Jul 12 17:57:22 mindpipe kernel: [do_journal_end+1853/2848] do_journal_end+0x73d/0xb20
Jul 12 17:57:22 mindpipe kernel: [journal_end_sync+54/112] journal_end_sync+0x36/0x70
Jul 12 17:57:22 mindpipe kernel: [__commit_trans_jl+223/240] __commit_trans_jl+0xdf/0xf0
Jul 12 17:57:22 mindpipe kernel: [reiserfs_commit_for_inode+32/64]
reiserfs_commit_for_inode+0x20/0x40
Jul 12 17:57:22 mindpipe kernel: [reiserfs_sync_file+77/144]
reiserfs_sync_file+0x4d/0x90
Jul 12 17:57:22 mindpipe kernel: [sys_fsync+123/176] sys_fsync+0x7b/0xb0
Jul 12 17:57:22 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Hopefully this is not too much information. You did ask for details...
Lee Revell
Mindpipe Audio
9:38 p.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
OK, thanks. The problem areas there are the timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
resierfs: yes, it's a problem. I "fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
What we need to do is to encourage audio testers
to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
OK, here goes. The following traces result from running JACK overnight
like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with
a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch.
voluntary_preempt and kernel_preemption are both on.
jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
--playback --period 32 --nperiods 2
These settings require less than 666 microseconds scheduler latency.
The average performance is quite good - 5-20 *microseconds*! 
10:08 p.m.
resierfs: yes, it's a problem. I "fixed"
it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?
--p
10:17 p.m.
Paul Davis <paul(a)linuxaudiosystems.com> wrote:
It seems that way, yes. I do not know how common the holdoffs are in real
life. It would be interesting if there was a user report that switching
from reiserfs to ext2/ext3 actually made a difference - this would tell us
that it is indeed a real-world problem.
Note that this info because available because someone set
/proc/asound/*/*/xrun_debug. We need more people doing that.
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?
10:32 p.m.
New subject: [linux-audio-dev] Re: Voluntary Kernel Preemption Patch
Andrew Morton writes:
Paul Davis <paul(a)linuxaudiosystems.com> wrote:
It seems that way, yes. I do not know how common the holdoffs are in real
life. It would be interesting if there was a user report that switching
from reiserfs to ext2/ext3 actually made a difference - this would tell us
that it is indeed a real-world problem.
Note that this info because available because someone set
/proc/asound/*/*/xrun_debug. We need more people doing that
Can I just point out that the reiserfs3.6 delays that I originally reported
with the preempt threshhold test did not come up once the patch was fixed. I
have my preempt threshold set at 1ms and did not see one single reiserfs
dump in my syslog. ie I don't think I am personally seeing any significant
reiserfs3.6 latencies.
Cheers,
Con
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?
11:11 p.m.
On Mon, 2004-07-12 at 20:06, Andrew Morton wrote:
Paul Davis <paul(a)linuxaudiosystems.com> wrote:
It seems that way, yes. I do not know how common the holdoffs are in real
life. It would be interesting if there was a user report that switching
from reiserfs to ext2/ext3 actually made a difference - this would tell us
that it is indeed a real-world problem.
This was not a synthetic benchmark, I would consider this a 'real-world'
problem now. Repeating the test with ext3 would just tell you whether
it has the same problem.
If it is neccesary to get the reiserfs issue addressed, I will repeat
the test with an ext3 system in the next few days, I would like to hear
from reiser on this before doing much more.
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?
Note that this info because available because someone
set
/proc/asound/*/*/xrun_debug. We need more people doing that.
-
This goes back to the need for ALSA documentation. Someone needs to
write some. This will probably require paying that person. Hopefully
SuSe is working on this, though I suspect I would have heard something.
To unsubscribe from this list: send the line
"unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
13 Jul
13 Jul
6 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
At Mon, 12 Jul 2004 21:01:16 -0400,
Lee Revell wrote:
Documentation/sound/alsa/Procfile.txt
Takashi
Note that this info because available because
someone set
/proc/asound/*/*/xrun_debug. We need more people doing that.
-
This goes back to the need for ALSA documentation. Someone needs to
write some. This will probably require paying that person. Hopefully
SuSe is working on this, though I suspect I would have heard something.
6:22 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Tue, 2004-07-13 at 03:49, Takashi Iwai wrote:
At Mon, 12 Jul 2004 21:01:16 -0400,
Lee Revell wrote:
Documentation/sound/alsa/Procfile.txt
I should have been more specific, I mean user-level documentation. All
the necessary information for developers is certainly out there, your
driver guide for example is excellent. I also didn't mean to imply that
the ALSA developers should do it, they are busy enough already...
Lee
Note that this info because available because
someone set
/proc/asound/*/*/xrun_debug. We need more people doing that.
-
This goes back to the need for ALSA documentation. Someone needs to
write some. This will probably require paying that person. Hopefully
SuSe is working on this, though I suspect I would have heard something. 
8:06 a.m.
On Mon, Jul 12, 2004 at 05:06:49 -0700, Andrew Morton wrote:
Paul Davis <paul(a)linuxaudiosystems.com> wrote:
It seems that way, yes. I do not know how common the holdoffs are in real
life. It would be interesting if there was a user report that switching
from reiserfs to ext2/ext3 actually made a difference - this would tell us
that it is indeed a real-world problem.
I think there was a report on linux-audio-user where switch from reiser to
ext[23] fixed someones latency problems.
- Steve
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
andrew, this is really helpful. should we conclude that until some
announcement from reiser that they have addressed this, the reiserfs
should be avoided on low latency systems?
12 Jul
12 Jul
10:11 p.m.
OK, thanks. The problem areas there are the
timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
i have one concern about this that i talked to takashi about when we
were in bordeaux. it seems to me that the ALSA xrun debug stuff is
measuring things when the interrupt handler for the ALSA device
executes and detects an xrun. if the handler itself was delayed, then
the stack trace for its execution doesn't or might not show what
caused the delay. this means, perhaps, that we need to be rather
careful interpreting these traces.
--p
10:19 p.m.
Paul Davis <paul(a)linuxaudiosystems.com> wrote:
We can usually tell that from the trace. If it points up into a busy piece
of code then it's pretty obvious what's happening. If the trace is due to
a long irq-off time then it will point up into the offending
local_irq_enable().
OK, thanks. The problem areas there are the
timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
i have one concern about this that i talked to takashi about when we
were in bordeaux. it seems to me that the ALSA xrun debug stuff is
measuring things when the interrupt handler for the ALSA device
executes and detects an xrun. if the handler itself was delayed, then
the stack trace for its execution doesn't or might not show what
caused the delay. this means, perhaps, that we need to be rather
careful interpreting these traces.
13 Jul
13 Jul
2:35 p.m.
On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew Morton wrote:
of code then it's pretty obvious what's
happening. If the trace is due to
a long irq-off time then it will point up into the offending
local_irq_enable().
schedule should be called with irq enabled, and I noticed here it's not
(jnz work_resched is executed with irq off so there is a window for
schedule to be called with irq off):
Index: linux-2.5/arch/i386/kernel/entry.S
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
retrieving revision 1.86
diff -u -p -r1.86 entry.S
--- linux-2.5/arch/i386/kernel/entry.S 23 May 2004 05:03:15 -0000 1.86
+++ linux-2.5/arch/i386/kernel/entry.S 13 Jul 2004 04:21:55 -0000
@@ -302,6 +310,7 @@ work_pending:
testb $_TIF_NEED_RESCHED, %cl
jz work_notifysig
work_resched:
+ sti
call schedule
cli # make sure we don't miss an interrupt
# setting need_resched or sigpending
4:52 p.m.
On Tue, 2004-07-13 at 12:25, Andrea Arcangeli wrote:
On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew
Morton wrote:
Would this explain these? When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc). I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
Jul 13 14:36:16 mindpipe kernel: [__crc_totalram_pages+1386981/5353478]
snd_pcm_period_elapsed+0x1af/0x410 [snd_pcm]
Jul 13 14:36:16 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 14:36:16 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 14:36:16 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 14:36:16 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel: [__do_softirq+48/144] __do_softirq+0x30/0x90
Jul 13 14:36:16 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 14:36:16 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 14:36:16 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel:
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -31, max jitter = 32): wrong interrupt acknowledge?
Jul 13 14:36:16 mindpipe kernel: [__crc_totalram_pages+1386981/5353478]
snd_pcm_period_elapsed+0x1af/0x410 [snd_pcm]
Jul 13 14:36:16 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 14:36:16 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 14:36:16 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 14:36:16 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 14:36:16 mindpipe kernel: [dnotify_parent+42/160] dnotify_parent+0x2a/0xa0
Jul 13 14:36:16 mindpipe kernel: [vfs_read+206/256] vfs_read+0xce/0x100
Jul 13 14:36:16 mindpipe kernel: [sys_read+45/80] sys_read+0x2d/0x50
Jul 13 14:36:16 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 14:36:16 mindpipe kernel:
Lee
of code then it's pretty obvious what's
happening. If the trace is due to
a long irq-off time then it will point up into the offending
local_irq_enable().
schedule should be called with irq enabled, and I noticed here it's not
(jnz work_resched is executed with irq off so there is a window for
schedule to be called with irq off):
Index: linux-2.5/arch/i386/kernel/entry.S
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
retrieving revision 1.86
diff -u -p -r1.86 entry.S
--- linux-2.5/arch/i386/kernel/entry.S 23 May 2004 05:03:15 -0000 1.86
+++ linux-2.5/arch/i386/kernel/entry.S 13 Jul 2004 04:21:55 -0000
@@ -302,6 +310,7 @@ work_pending:
testb $_TIF_NEED_RESCHED, %cl
jz work_notifysig
work_resched:
+ sti
call schedule
cli # make sure we don't miss an interrupt
# setting need_resched or sigpending
7:36 p.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Would this explain these? When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc). I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
I'm wondering what this message actually means. "Unexpected hw_pointer
value"?
Does this actually indicate an underrun, or is the debug code screwy?
7:55 p.m.
On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Not sure. Here is what Takashi had to say about it:
"The message appears when an unexpected DMA pointer is read in the
interrupt handler. Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.
Since you're using quite small buffer, I guess the former case."
My response:
"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size. Sometimes these messages are
associated with XRUNs, sometimes not."
Haven't heard back yet.
Is it possible that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
Lee
Would this explain these? When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc). I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
I'm wondering what this message actually means. "Unexpected hw_pointer
value"?
Does this actually indicate an underrun, or is the debug code screwy?
14 Jul
14 Jul
7:01 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
I think yes. 32 frames / 44.1kHz = 0.725 ms.
Takashi
Lee Revell <rlrevell(a)joe-job.com> wrote:
Not sure. Here is what Takashi had to say about it:
"The message appears when an unexpected DMA pointer is read in the
interrupt handler. Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.
Since you're using quite small buffer, I guess the former case."
My response:
"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size. Sometimes these messages are
associated with XRUNs, sometimes not."
Haven't heard back yet.
Is it possible that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
Would this explain these? When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc). I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
I'm wondering what this message actually means. "Unexpected hw_pointer
value"?
Does this actually indicate an underrun, or is the debug code screwy?
7:46 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
I am runnign at 48kHz so it's actually 0.666 ms. But, the average
response is quite good, 20-30 microseconds. The spikes are infrequent
enought that I think this is achievable. If not then 64 frames
definitely is.
So what is the difference between the above message and an XRUN? I
thought an XRUN occurred when the handling of the IRQ is delayed more
than the buffer size.
Lee
On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
I think yes. 32 frames / 44.1kHz = 0.725 ms.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Not sure. Here is what Takashi had to say about it:
"The message appears when an unexpected DMA pointer is read in the
interrupt handler. Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.
Since you're using quite small buffer, I guess the former case."
My response:
"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size. Sometimes these messages are
associated with XRUNs, sometimes not."
Haven't heard back yet.
Is it possible that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
Would this explain these? When running JACK with settings that need
sub-millisecond latencies, I get them when I generate any load at all on
the system (typing, switching windows, etc). I also get lots of these
if I run JACK from an X terminal, but very few if I run it from a text
console, even if X is running in the background.
Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
I'm wondering what this message actually means. "Unexpected hw_pointer
value"?
Does this actually indicate an underrun, or is the debug code screwy?
8:58 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
At Wed, 14 Jul 2004 05:36:31 -0400,
Lee Revell wrote:
On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
Yes, XRUN is so.
The message above is not directly related with XRUN.
It appears when an unexpected DMA pointer is returned. This can
happen e.g. when the irq handler is called at the wrong timing,
or the register returns a bogus value.
Takashi
At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
I am runnign at 48kHz so it's actually 0.666 ms. But, the average
response is quite good, 20-30 microseconds. The spikes are infrequent
enought that I think this is achievable. If not then 64 frames
definitely is.
So what is the difference between the above message and an XRUN? I
thought an XRUN occurred when the handling of the IRQ is delayed more
than the buffer size.
On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
I think yes. 32 frames / 44.1kHz = 0.725 ms.
Lee Revell <rlrevell(a)joe-job.com> wrote:
>
> Would this explain these? When running JACK with settings that need
> sub-millisecond latencies, I get them when I generate any load at all on
> the system (typing, switching windows, etc). I also get lots of these
> if I run JACK from an X terminal, but very few if I run it from a text
> console, even if X is running in the background.
>
> Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
I'm wondering what this message actually means. "Unexpected hw_pointer
value"?
Does this actually indicate an underrun, or is the debug code screwy?
Not sure. Here is what Takashi had to say about it:
"The message appears when an unexpected DMA pointer is read in the
interrupt handler. Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.
Since you're using quite small buffer, I guess the former case."
My response:
"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size. Sometimes these messages are
associated with XRUNs, sometimes not."
Haven't heard back yet.
Is it possible that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
1:02 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Wed, 2004-07-14 at 06:48, Takashi Iwai wrote:
At Wed, 14 Jul 2004 05:36:31 -0400,
Lee Revell wrote:
I am still kind of confused. It sounds like this is more serious than
an XRUN. What would cause this to happen?
Lee
On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
Yes, XRUN is so.
The message above is not directly related with XRUN.
It appears when an unexpected DMA pointer is returned. This can
happen e.g. when the irq handler is called at the wrong timing,
or the register returns a bogus value.
At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
I am runnign at 48kHz so it's actually 0.666 ms. But, the average
response is quite good, 20-30 microseconds. The spikes are infrequent
enought that I think this is achievable. If not then 64 frames
definitely is.
So what is the difference between the above essage and an XRUN? I
thought an XRUN occurred when the handling of the IRQ is delayed more
than the buffer size.
On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> Lee Revell <rlrevell(a)joe-job.com> wrote:
> >
> > Would this explain these? When running JACK with settings that need
> > sub-millisecond latencies, I get them when I generate any load at all on
> > the system (typing, switching windows, etc). I also get lots of these
> > if I run JACK from an X terminal, but very few if I run it from a text
> > console, even if X is running in the background.
> >
> > Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
>
> I'm wondering what this message actually means. "Unexpected hw_pointer
> value"?
>
> Does this actually indicate an underrun, or is the debug code screwy?
Not sure. Here is what Takashi had to say about it:
"The message appears when an unexpected DMA pointer is read in the
interrupt handler. Either the handling of irq was delayed more than
the buffer size, an irq is issued at the wrong timing, or the DMA
pointer reigster is somehow screwed up.
Since you're using quite small buffer, I guess the former case."
My response:
"I thought this was what an XRUN was, when the handling of the irq is
delayed more than the buffer size. Sometimes these messages are
associated with XRUNs, sometimes not."
Haven't heard back yet.
Is it possible that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
I think yes. 32 frames / 44.1kHz = 0.725 ms.
2:39 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Wed, 2004-07-14 at 10:52, Lee Revell wrote:
On Wed, 2004-07-14 at 06:48, Takashi Iwai wrote:
OK, I think I understand the distinction. An XRUN is when a process is
writing (or reading) to a PCM device, the buffer half empty interrupt
occurs, and the interrupt handler runs, which makes the process ready to
run, but the scheduler does not run it in time to write the next block
of data to the buffer.
The "Unexpected hw_pointer" error would occur when the device generates
an interrupt, but there is a delay in running the interrupt handler, and
by the time the interrupt handler runs, the hardware pointer has moved
somewhere we don't expect it to be. This would occur if we missed an
interrupt, or several, due to interrupts being disabled for longer than
the period size. So the distinction is scheduler latency vs. interrupt
response latency.
What is the longest that the kernel disables interrupts? I remember
hearing 500 microseconds, but this was a while back.
Lee
At Wed, 14 Jul 2004 05:36:31 -0400,
Lee Revell wrote:
I am still kind of confused. It sounds like this is more serious than
an XRUN. What would cause this to happen?
On Wed, 2004-07-14 at 04:51, Takashi Iwai wrote:
Yes, XRUN is so.
The message above is not directly related with XRUN.
It appears when an unexpected DMA pointer is returned. This can
happen e.g. when the irq handler is called at the wrong timing,
or the register returns a bogus value.
At Tue, 13 Jul 2004 17:45:30 -0400,
Lee Revell wrote:
>
> On Tue, 2004-07-13 at 17:29, Andrew Morton wrote:
> > Lee Revell <rlrevell(a)joe-job.com> wrote:
> > >
> > > Would this explain these? When running JACK with settings that need
> > > sub-millisecond latencies, I get them when I generate any load at all on
> > > the system (typing, switching windows, etc). I also get lots of these
> > > if I run JACK from an X terminal, but very few if I run it from a text
> > > console, even if X is running in the background.
> > >
> > > Jul 13 14:36:16 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:199: Unexpected hw_pointer
value [1] (stream = 0, delta: -25, max jitter = 32): wrong interrupt acknowledge?
> >
> > I'm wondering what this message actually means. "Unexpected
hw_pointer
> > value"?
> >
> > Does this actually indicate an underrun, or is the debug code screwy?
>
> Not sure. Here is what Takashi had to say about it:
>
> "The message appears when an unexpected DMA pointer is read in the
> interrupt handler. Either the handling of irq was delayed more than
> the buffer size, an irq is issued at the wrong timing, or the DMA
> pointer reigster is somehow screwed up.
>
> Since you're using quite small buffer, I guess the former case."
>
> My response:
>
> "I thought this was what an XRUN was, when the handling of the irq is
> delayed more than the buffer size. Sometimes these messages are
> associated with XRUNs, sometimes not."
>
> Haven't heard back yet.
>
> Is it possible that I am simply pushing my hardware past its limits?
> Keep in mind this is a 600Mhz C3 processor.
I think yes. 32 frames / 44.1kHz = 0.725 ms.
I am runnign at 48kHz so it's actually 0.666 ms. But, the average
response is quite good, 20-30 microseconds. The spikes are infrequent
enought that I think this is achievable. If not then 64 frames
definitely is.
So what is the difference between the above essage and an XRUN? I
thought an XRUN occurred when the handling of the IRQ is delayed more
than the buffer size. 
12:35 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Takashi Iwai wrote:
I don't think so, I think it's because the Linux scheduler (and kernel
in general) since it's not a RTOS
is pushed to the limits. (but as we see it can still be optimized).
For example I used the same VIA box with RTLinux to remote control servo
motors which need a PWM signal
of the duration of 2msec and based on the location of the negative flank
(from high to low) the servo
motor goes in a certain position.
For example if the duration of the pulse is 2msec then setting the flank
at 0msec (at the beginning of the cycle)
the servo goes to -45degrees , 1msec 0degrees , 2msec +45degrees.
Jitter in the pulse can be detected when the servo is vibrating a bit
around the nominal position.
Of course a very short lived spike cannot be detected by the servo
because of the motor's inertia
but I tried to put the box under very high load especially video
playback (the VIA box uses a shared bus architecture
holding the video data in the PC's RAM), HD load etc but the jitter is
very minimal, probably 30-40usec because the
servos vibrate only about 1degree or so (only when the box is under very
high load).
This just to say that the VIA box should easily be able to cope with 32
audio frames (0.6msec buffers) from a hardware
point of view.
Anyway "worst case" latencies (or better latencies under very high load)
of <0.5-1msec are completely adequate for
real time multimedia because if you shorten your audio processing cycles
too much (eg 32 frames) then the setup
overhead of DSP routines, and scheduling overhead becomes big and the
efficiency of
the algorithms decrease quite a bit.
imagine running jack with 10 client applications and 32 frames (0.6msec
periods) , this means that within 0.6msec you need
to reschedule 10 times = 60 usec per client. I don't know how much the
actual rescheduling of a process takes (AFAIK around 1usec ?)
but I guess the main problem is the constant invalidation of the cache
because those audio clients run for a really short time.
Of course if you have only 1-2 clients running then the efficiency at 32
frames should still be good (but it will certainly provide 10-20% less
perfromance than using 64 or 128 frames).
Our goal should be rock solid operation at 64msec (around 1msec-1.5msec
processing periods).
If you use 2-3 buffers then the kernel has still another 1.5-3msec of
headroom before an actual (audible) xrun occurs.
cheers,
Benno
http://www.linuxsampler.org
Is it possible
that I am simply pushing my hardware past its limits?
Keep in mind this is a 600Mhz C3 processor.
I think yes. 32 frames / 44.1kHz = 0.725 ms.
Takashi
13 Jul
13 Jul
4:59 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
On Mon, Jul 12, 2004 at 05:08:44PM -0700, Andrew Morton wrote:
sys_sched_yield() also calls schedule() with local interrupts disabled.
It's a bit grubby, but saves a few cycles. Nick and Ingo prefer it that way.
of code then it's pretty obvious what's
happening. If the trace is due to
a long irq-off time then it will point up into the offending
local_irq_enable().
schedule should be called with irq enabled, and I noticed here it's not
(jnz work_resched is executed with irq off so there is a window for
schedule to be called with irq off):
Index: linux-2.5/arch/i386/kernel/entry.S
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/arch/i386/kernel/entry.S,v
retrieving revision 1.86
diff -u -p -r1.86 entry.S
--- linux-2.5/arch/i386/kernel/entry.S 23 May 2004 05:03:15 -0000 1.86
+++ linux-2.5/arch/i386/kernel/entry.S 13 Jul 2004 04:21:55 -0000
@@ -302,6 +310,7 @@ work_pending:
testb $_TIF_NEED_RESCHED, %cl
jz work_notifysig
work_resched:
+ sti
call schedule
cli # make sure we don't miss an interrupt
# setting need_resched or sigpending
7:48 p.m.
On Tue, Jul 13, 2004 at 11:48:29AM -0700, Andrew Morton wrote:
sys_sched_yield() also calls schedule() with local
interrupts disabled.
It's a bit grubby, but saves a few cycles. Nick and Ingo prefer it that way.
we can remove the irqs_disabled() check in might_sleep then, I'd like to
call might_sleep from cond_resched.
8:01 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
On Tue, Jul 13, 2004 at 11:48:29AM -0700, Andrew Morton wrote:
Confused. Where do we call cond_resched() with local interrupts disabled?
Sleeping with local interrupts disabled is usually a bug, so we should prefer
to keep that check in might_sleep().
sys_sched_yield() also calls schedule() with
local interrupts disabled.
It's a bit grubby, but saves a few cycles. Nick and Ingo prefer it that way.
we can remove the irqs_disabled() check in might_sleep then, I'd like to
call might_sleep from cond_resched.
8:11 p.m.
On Tue, Jul 13, 2004 at 02:54:24PM -0700, Andrew Morton wrote:
Confused. Where do we call cond_resched() with local
interrupts disabled?
there are a lot of cond_resched, we might be calling it with irq
disabled, nobody ever did a might_sleep in the fast path of
cond_resched. And even if nobody does, if entry.S and sched_yield can
optimize, then everybody else can optimize too. It's not like that
entry.S is a piece of scheduler internal that will be obviously modified
if we modify the scheduler. so if you intend to leave those two there's
no point to forbid others to optimize, and there's no might_sleep in
current cond_resched anyways so you're already allowing people to
optimize and I want to allow it still after I add might_sleep there.
Sleeping with local interrupts disabled is usually a
bug, so we should
prefer to keep that check in might_sleep().
either it's _always_ a bug including for entry.S or sched_yield, or it's
_never_ a bug. I don't understand the "usually".
8:32 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
If some code does:
local_irq_disable();
<fiddle with per-cpu stuff>
kmalloc(GFP_KERNEL);
<fiddle with per-cpu stuff>
local_irq_enable();
or
local_irq_disable();
<fiddle with per-cpu stuff>
function_which_calls_cond_resched();
<fiddle with per-cpu stuff>
local_irq_enable();
then we want might_sleep() to warn about the bug.
The fact that a couple of scheduler-internal fastpaths happen to know that
they can call schedule() with interrupts disabled is not relevant to this.
Sleeping with local interrupts disabled is
usually a bug, so we should
prefer to keep that check in might_sleep().
either it's _always_ a bug including for entry.S or sched_yield, or it's
_never_ a bug. I don't understand the "usually".
8:47 p.m.
On Tue, Jul 13, 2004 at 03:25:32PM -0700, Andrew Morton wrote:
local_irq_disable();
<fiddle with per-cpu stuff>
function_which_calls_cond_resched();
<fiddle with per-cpu stuff>
local_irq_enable();
then we want might_sleep() to warn about the bug.
might_sleep currently _doesn't_ warn about any bug in the above case I
quoted.
the kmalloc example is trapped instead.
From my part I don't like anybody to call schedule
with irq disabled
(and I would definitely put a debug check in schedule() for that,
guess
how I found about the missing sti in entry.S btw). But if you are ok
with people calling schedule with irq disabled then I cannot put a check
in there. sti doesn't cost that much, and the work-to-do and sched_yield
paths are _never_ fast paths, so they don't worth an hack like that.
8:51 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
On Tue, Jul 13, 2004 at 03:25:32PM -0700, Andrew Morton wrote:
Yeah, I know. might_sleep() in cond_resched() makes sense.
local_irq_disable();
<fiddle with per-cpu stuff>
function_which_calls_cond_resched();
<fiddle with per-cpu stuff>
local_irq_enable();
then we want might_sleep() to warn about the bug.
might_sleep currently _doesn't_ warn about any bug in the above case I
quoted.
the kmalloc example is trapped instead. >From my part I don't like anybody to call
schedule with irq disabled
I'd agree with that. But when I "fixed" it, Ingo unfixed it again and I
didn't have sufficiently strong feelings either way to object.
9:03 p.m.
On Tue, Jul 13, 2004 at 03:44:48PM -0700, Andrew Morton wrote:
Yeah, I know. might_sleep() in cond_resched() makes
sense.
What I'm doing is basically to replace all might_sleep with cond_resched
and then I add might_sleep in cond_resched. I also merged all
new might_sleep in Ingo's patch (converted to cond_resched). We'll see
what happens then when I try to boot such a thing (the sti and
sched_yield already given me some troubles).
I was considering adding a cond_resched_costly but I didn't see anything
really that costly to need a CONFIG_LOW_RESCHED_OVERHEAD.
btw, cond_resched should only be defined as might_sleep with PREEMPT
enabled, otherwise it's pointless to check need_resched at almost every
spin_unlock and to do it during cond_resched too. if might_sleep doesn't
BUG it means we didn't need to check need_resched in the first place if
preempt is enabled.
cond_resched_lock is another story of course.
9:13 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
What I'm doing is basically to replace all might_sleep with cond_resched
I cannot see a lot of point in that. They are semantically different
things and should look different in the source.
And it's currently OK to add a might_sleep() to (say) an inline path which
is expended a zillion times because we know it'll go away for production
builds. If those things become cond_resched() calls instead, the code
increase will be permanent.
cond_resched_lock is another story of course.
cond_resched_lock() doesn't work on SMP. I'll probably be removing it in
favour of unconditionally dropping the lock every N times around the loop,
to allow the other CPU (the one with need_resched() true) to get in there
and take it.
And please let me repeat: preemption is the way in which we wish to provide
low-latency. At this time, patches which sprinkle cond_resched() all over
the place are unwelcome. After 2.7 forks we can look at it again.
I've yet to go through Arjan's patch - I suspect a lot of it is not needed.
9:28 p.m.
On Tue, Jul 13, 2004 at 04:06:28PM -0700, Andrew Morton wrote:
I think it's a mistake to believe people that leaves preempt off don't
care about lowlatency (this is also why I always keep this change in my
tree)
Index: linux-2.5/mm/memory.c
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/mm/memory.c,v
retrieving revision 1.178
diff -u -p -r1.178 memory.c
--- linux-2.5/mm/memory.c 30 Jun 2004 02:21:13 -0000 1.178
+++ linux-2.5/mm/memory.c 13 Jul 2004 03:33:45 -0000
@@ -479,20 +479,13 @@ static void unmap_page_range(struct mmu_
}
/* Dispose of an entire struct mmu_gather per rescheduling point */
-#if defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
+#if defined(CONFIG_SMP)
#define ZAP_BLOCK_SIZE (FREE_PTE_NR * PAGE_SIZE)
-#endif
-
+#else
/* For UP, 256 pages at a time gives nice low latency */
-#if !defined(CONFIG_SMP) && defined(CONFIG_PREEMPT)
#define ZAP_BLOCK_SIZE (256 * PAGE_SIZE)
#endif
-/* No preempt: go for improved straight-line efficiency */
-#if !defined(CONFIG_PREEMPT)
-#define ZAP_BLOCK_SIZE (1024 * PAGE_SIZE)
-#endif
-
/**
* unmap_vmas - unmap a range of memory covered by a list of vma's
* @tlbp: address of the caller's struct mmu_gather
having a CONFIG_PREEMPT in the above code, means you believe the people
who leaves preempt off don't care about lowlatency and that's really not
true. Ingo's effort as well shows people care about lowlatency even with
preempt off.
The people who leaves preempt off simply thinks they can get the same
lowlatency but with less overhead in the locks (though there will be
cond_resched in the semaphore, but semaphore is less performance
critical than the spinlock normally, and semaphore is usually hold for a
longer period of time, short critical sections would better go with the
spinlock anyways to avoid overscheduling).
Andrea Arcangeli <andrea(a)suse.de> wrote:
this is exactly why I'm making this change: so you can still add
might_sleep in a inline path expected to run a zillion times. With
Ingo's change you would be doing cond_sched internally to might_sleep, I
do the other way around so might_sleep remains a debugging statement.
As Ingo basically showed (and I agree), all current might_sleep seems
suitable to be converted to cond_resched. I checked all them and there's
none that seems to be called in a loop for no good reason. the ones in
the semaphore are quite nice too since it's better to schedule right
before taking the semaphore than after getting it. The one in the
semaphore and in the copy-user are the only place where preempt seems to
be lower overhead but in most other places a spinlock is being taken
very near to the cond_resched.
What I'm doing is basically to replace all might_sleep with cond_resched
I cannot see a lot of point in that. They are semantically different
things and should look different in the source.
And it's currently OK to add a might_sleep() to (say) an inline path which
is expended a zillion times because we know it'll go away for production
builds. If those things become cond_resched() calls instead, the code
increase will be permanent.
cond_resched_lock is another story of course.
cond_resched_lock() doesn't work on SMP. I'll probably be removing it in
favour of unconditionally dropping the lock every N times around the loop,
to allow the other CPU (the one with need_resched() true) to get in there
and take it.
And please let me repeat: preemption is the way in which we wish to provide
low-latency. At this time, patches which sprinkle cond_resched() all over
the place are unwelcome. After 2.7 forks we can look at it again. I've yet to go through Arjan's patch - I
suspect a lot of it is not needed.
Arjan's or Ingo's? I've seen Ingo's patch but maybe I missed Arjan's
one.
9:39 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
OK.
cond_resched() is usually a waste of space with CONFIG_PREEMPT. It might
make sense to define a cond_resched_if_not_preempt thingy, which only does
things if !CONFIG_PREEMPT. We'd still need to use cond_resched() inside
lock_kernel().
I think Ingo's patch includes Arjan's one. Ingo's patch apparently breaks
ext3. I have a bunch of ext3 and other fixes here, but there's still an occasional
problem on SMP.
And it's currently OK to add a might_sleep()
to (say) an inline path which
is expended a zillion times because we know it'll go away for production
builds. If those things become cond_resched() calls instead, the code
increase will be permanent.
this is exactly why I'm making this change: so you can still add
might_sleep in a inline path expected to run a zillion times. With
Ingo's change you would be doing cond_sched internally to might_sleep, I
do the other way around so might_sleep remains a debugging statement. I've yet to go through Arjan's patch - I
suspect a lot of it is not needed.
Arjan's or Ingo's? I've seen Ingo's patch but maybe I missed Arjan's
one.
10:50 p.m.
On Tue, Jul 13, 2004 at 04:32:36PM -0700, Andrew Morton wrote:
OK.
cond_resched() is usually a waste of space with CONFIG_PREEMPT. It might
make sense to define a cond_resched_if_not_preempt thingy, which only does
things if !CONFIG_PREEMPT. We'd still need to use cond_resched() inside
lock_kernel().
Though most of the current cond_resched invocations in mainline (and the
future ones too) should be optimized away with preempt enabled, so maybe
it's simpler to just implement a cond_resched_force() that isn't
optimized away with preempt enabled and use it in lock_kernel? I'm
missing where cond_resched is needed inside lock_kernel though,
preempt_schedule already checks TIF_NEED_RESCHED by hand (on the top of
my mind I don't recall cond_rescheds that shouldn't be optimized away
with preempt enabled).
I think Ingo's patch includes Arjan's one.
Ingo's patch apparently breaks
Ok.
ext3. I have a bunch of ext3 and other fixes here,
but there's still an occasional
problem on SMP.
I also had instability with it and I've seen debug checks in ext3 too
(but it's not finished anyways), and I've still a bit of printk flooding
so I'm uncertain if it's my mistake due the more strict debugging I
added in schedule() or an issue in the newer cond_resched but thanks for
the info (sounds like not all the debug checks triggering are my mistake ;).
19 Jul
19 Jul
10:19 a.m.
* Andrea Arcangeli <andrea(a)suse.de> wrote:
As Ingo basically showed (and I agree), all current
might_sleep seems
suitable to be converted to cond_resched. I checked all them and
there's none that seems to be called in a loop for no good reason. the
ones in the semaphore are quite nice too since it's better to schedule
right before taking the semaphore than after getting it. The one in
the semaphore and in the copy-user are the only place where preempt
seems to be lower overhead but in most other places a spinlock is
being taken very near to the cond_resched.
yes. Btw., i'm not sure whether you've noticed but last week i've also
created a 'clean' variant of the patch. The latest version against -mm
is:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-clean-2.6.7-mm…
this one doesnt have any of the debugging/development helpers and
switches. I have still made it a .config option. Note how minimal the
patch became this way.
Ingo
2 Sep
2 Sep
8:20 p.m.
Hi Ingo,
On Mon, Jul 19, 2004 at 01:59:52PM +0200, Ingo Molnar wrote:
yes. Btw., i'm not sure whether you've noticed
but last week i've also
created a 'clean' variant of the patch. The latest version against -mm
is:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-clean-2.6.7-mm…
this one doesnt have any of the debugging/development helpers and
switches. I have still made it a .config option. Note how minimal the
patch became this way.
as said in a part of a previous email might_sleep() just like BUG() can
be defined to noop.
cond_resched() is the API to use.
if you're scared that there are too many cond_resched (I'm not scared
and people should enable them anyways if they make a difference, they
still should be less than the number of spin_unlocks with preempt
enabled), then you should add a cond_resched_costly and add a config
option that turns it off. I think you don't even need to add the config
option, you can define cond_resched_costly as cond_resched, and to just
use it to mark the places that might be expensive.
Then you should change cond_resched to call might_sleep in the else
branch (as I discussed with Andrew last month).
this was the core of the patch I was playing with last month which
should be still valid and it solves the preprocessor issues with
cond_resched (and I hope the bug was not in the below code ;)
could you modify your patch accordingly? thanks!
Index: linux-2.5/include/linux/kernel.h
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/include/linux/kernel.h,v
retrieving revision 1.53
diff -u -p -r1.53 kernel.h
--- linux-2.5/include/linux/kernel.h 27 Jun 2004 17:55:19 -0000 1.53
+++ linux-2.5/include/linux/kernel.h 13 Jul 2004 02:19:43 -0000
@@ -48,12 +48,23 @@ struct completion;
#ifdef CONFIG_DEBUG_SPINLOCK_SLEEP
void __might_sleep(char *file, int line);
#define might_sleep() __might_sleep(__FILE__, __LINE__)
-#define might_sleep_if(cond) do { if (unlikely(cond)) might_sleep(); } while (0)
#else
#define might_sleep() do {} while(0)
-#define might_sleep_if(cond) do {} while (0)
#endif
+#define need_resched() unlikely(test_thread_flag(TIF_NEED_RESCHED))
+
+extern void __cond_resched(void);
+#define cond_resched() \
+do { \
+ if (need_resched()) \
+ __cond_resched(); \
+ else \
+ might_sleep(); \
+} while (0)
+
+#define cond_resched_if(cond) do { if (unlikely(cond)) cond_resched(); } while (0)
+
#define abs(x) ({ \
int __x = (x); \
(__x < 0) ? -__x : __x; \
Index: linux-2.5/include/linux/sched.h
===================================================================
RCS file: /home/andrea/crypto/cvs/linux-2.5/include/linux/sched.h,v
retrieving revision 1.245
diff -u -p -r1.245 sched.h
--- linux-2.5/include/linux/sched.h 2 Jul 2004 17:31:23 -0000 1.245
+++ linux-2.5/include/linux/sched.h 13 Jul 2004 02:33:12 -0000
@@ -1012,18 +1013,6 @@ static inline int signal_pending(struct
{
return unlikely(test_tsk_thread_flag(p,TIF_SIGPENDING));
}
-
-static inline int need_resched(void)
-{
- return unlikely(test_thread_flag(TIF_NEED_RESCHED));
-}
-
-extern void __cond_resched(void);
-static inline void cond_resched(void)
-{
- if (need_resched())
- __cond_resched();
-}
/*
* cond_resched_lock() - if a reschedule is pending, drop the given lock,
8:39 p.m.
Andrea Arcangeli <andrea(a)suse.de> wrote:
if you're scared that there are too many cond_resched (I'm not scared
and people should enable them anyways if they make a difference, they
still should be less than the number of spin_unlocks with preempt
enabled), then you should add a cond_resched_costly and add a config
option that turns it off.
None of these approaches improves worst-case latency at all on SMP. If
we're not going to address the SMP problem we could just make it UP-only,
in which case increased locking costs are a non-issue.
I'd prefer that we find a solution for SMP too though.
9:18 p.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
None of these approaches improves worst-case latency
at all on SMP.
If we're not going to address the SMP problem we could just make it
UP-only, in which case increased locking costs are a non-issue.
I'd prefer that we find a solution for SMP too though.
i have solved the fundamental SMP latency problems in the -Q7 patch, by
redesigning how SMP preemption is done. Here's the relevant changelog
entry:
[...]
the main change in this patch are more SMP latency fixes. The stock
kernel, even with CONFIG_PREEMPT enabled, didnt have any spin-nicely
preemption logic for the following, commonly used SMP locking
primitives: read_lock(), spin_lock_irqsave(), spin_lock_irq(),
spin_lock_bh(), read_lock_irqsave(), read_lock_irq(), read_lock_bh(),
write_lock_irqsave(), write_lock_irq(), write_lock_bh(). Only
spin_lock() and write_lock() [the two simplest cases] where covered.
In addition to the preemption latency problems, the _irq() variants in
the above list didnt do any IRQ-enabling while spinning - possibly
resulting in excessive irqs-off sections of code!
-Q7 fixes all of these latency problems: we now re-enable interrupts
while spinning in all possible cases, and a spinning op stays
preemptible if this is a beginning of a new critical section.
[...]
feedback from Mark H Johnson:
http://lkml.org/lkml/2004/8/30/202
the latest patch is:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.9-rc1-bk4-…
i'm already in the process of cleaning up the patch and making it ready
for splitup & merge. The spinlock fixes will be amongst the first
patches i'll send you.
Ingo
9:36 p.m.
* Ingo Molnar <mingo(a)elte.hu> wrote:
i have solved the fundamental SMP latency problems in
the -Q7 patch,
by redesigning how SMP preemption is done. Here's the relevant
changelog entry:
also this changelog explains the core changes that enable good
preemption latencies on SMP:
[...]
i took another look at SMP latencies, the last larger chunk of code that
produced millisec-category latencies. CONFIG_PREEMPT tries to solve some
of the SMP issues but there were still lots of problems remaining: main
problem area is spinlocks nested at multiple levels. If a piece of code
(e.g. the MM or ext3's journalling code) does the following:
spin_lock(&spinlock_1);
...
spin_lock(&spinlock_2);
...
then even with CONFIG_PREEMPT enabled, current kernels may spin on
spinlock_2 indefinitely. A number of critical sections break their long
paths by using cond_resched_lock(), but this does not break the path on
SMP, because need_resched() is not set in the above case.
(The -mm kernel introduced a couple of patches that try to drop
spinlocks unconditionally at a high frequency: but besides being a
kludge it's also a performance problem, we keep
dropping/waiting/retaking locks quite frequently. That solution also
doesnt solve the problem of cond_resched_lock() not working on SMP.)
to solve the problem i've introduced a new spinlock field,
lock->break_lock, which signals towards the holding CPU that a
spinlock-break is requested by another CPU. This field is only set if a
CPU is spinning in __preempt_spin_lock [at any locking depth], so the
default overhead is zero. I've extended cond_resched_lock() to check for
this flag - in this case we can also save a reschedule. I've added the
lock_need_resched(lock) and need_lockbreak(lock) methods to check for
the need to break out of a critical section.
preliminary results on a dual x86 box show a dramatic reduction in
latencies on SMP - where there used to be 5-10 msec latencies there are
close-to-UP latencies now. But it needs more testing.
[...]
Ingo
13 Jul
13 Jul
6:08 a.m.
At Mon, 12 Jul 2004 20:01:51 -0400,
Paul Davis wrote:
Well, it can catch up too long critical sections in most cases,
although it cannot measure the latency to wake up the audio thread.
So, the detection can't be 100% guaranteed.
Takashi
OK, thanks. The problem areas there are the
timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
i have one concern about this that i talked to takashi about when we
were in bordeaux. it seems to me that the ALSA xrun debug stuff is
measuring things when the interrupt handler for the ALSA device
executes and detects an xrun. if the handler itself was delayed, then
the stack trace for its execution doesn't or might not show what
caused the delay. this means, perhaps, that we need to be rather
careful interpreting these traces.
12 Jul
12 Jul
10:26 p.m.
On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Gnutella client. Forgot about that. I agree, it is not reasonable to
expect low latency with this kind of network traffic happening. I am
impressed it worked as well as it did.
On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
OK, thanks. The problem areas there are the timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
What we need to do is to encourage audio testers
to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
OK, here goes. The following traces result from running JACK overnight
like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with
a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch.
voluntary_preempt and kernel_preemption are both on.
jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
--playback --period 32 --nperiods 2
These settings require less than 666 microseconds scheduler latency.
The average performance is quite good - 5-20 *microseconds*! resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
Interesting. There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work. Should I try
ext3?
Lee
10:57 p.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
It seems to be misplaced. A simple make-a-zillion-teeny-files test here
exhibits a 14 millisecond holdoff.
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
Interesting. There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work. Should I try ext3?
ext3 is certainly better than that, but still has a couple of potential
problem spots. ext2 is probably the best at this time.
13 Jul
13 Jul
1:02 a.m.
On Mon, 2004-07-12 at 20:46, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
It seems to be misplaced. A simple make-a-zillion-teeny-files test here
exhibits a 14 millisecond holdoff.
Here are a few more:
Jul 12 22:20:41 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:41 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:41 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:41 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:41 mindpipe kernel: [unix_stream_recvmsg+135/1088]
unix_stream_recvmsg+0x87/0x440
Jul 12 22:20:41 mindpipe kernel: [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 22:20:41 mindpipe kernel: [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
Jul 12 22:20:41 mindpipe kernel: [do_sync_read+125/192] do_sync_read+0x7d/0xc0
Jul 12 22:20:41 mindpipe kernel: [do_select+435/752] do_select+0x1b3/0x2f0
Jul 12 22:20:41 mindpipe kernel: [vfs_read+223/256] vfs_read+0xdf/0x100
Jul 12 22:20:42 mindpipe kernel: [sys_read+45/80] sys_read+0x2d/0x50
Jul 12 22:20:42 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 22:20:42 mindpipe kernel:
Jul 12 22:20:44 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:44 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:44 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:44 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:44 mindpipe kernel:
Jul 12 22:20:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:49 mindpipe kernel:
Lee
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
Interesting. There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work. Should I try ext3?
ext3 is certainly better than that, but still has a couple of potential
problem spots. ext2 is probably the best at this time.
2:10 a.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Here are a few more:
Jul 12 22:20:41 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:41 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:41 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:41 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:41 mindpipe kernel: [unix_stream_recvmsg+135/1088]
unix_stream_recvmsg+0x87/0x440
Jul 12 22:20:41 mindpipe kernel: [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 22:20:41 mindpipe kernel: [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
Jul 12 22:20:41 mindpipe kernel: [do_sync_read+125/192] do_sync_read+0x7d/0xc0
Jul 12 22:20:41 mindpipe kernel: [do_select+435/752] do_select+0x1b3/0x2f0
Jul 12 22:20:41 mindpipe kernel: [vfs_read+223/256] vfs_read+0xdf/0x100
Jul 12 22:20:42 mindpipe kernel: [sys_read+45/80] sys_read+0x2d/0x50
Jul 12 22:20:42 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 22:20:42 mindpipe kernel:
Jul 12 22:20:44 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:44 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:44 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:44 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Oh dear, these don't make much sense.
You'll have the best chance of getting decent traces with
CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
will help a lot. Are you sure that CONFIG_PREEMPT is enabled on that
kernel?
6:07 a.m.
On Mon, 2004-07-12 at 23:59, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Yes:
root@mindpipe:/home/rlrevell/kernel-source/kernel-source-2.6.7# grep .
/proc/sys/kernel/*_preemption
/proc/sys/kernel/kernel_preemption:1
/proc/sys/kernel/voluntary_preemption:1
My testing also uncovered a possible bug in the emu10k1 driver, I am
looking into this. I don't think it's related because the log entries
are spaced minutes apart. Could this be a driver problem? Hardware?
Just as a reference point, what do you think is the longest delay I
*should* be seeing? I recall hearing that BEOS guaranteed that
interrupts are never disabled for more than 50 usecs. This seems
achievable, as the average delay I am seeing is 15 usecs.
I will set CONFIG_FRAME_POINTER=Y and CONFIG_4KSTACKS=n and test some
more.
Lee
Here are a few more:
Jul 12 22:20:41 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:41 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:41 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:41 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:41 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 12 22:20:41 mindpipe kernel: [unix_stream_recvmsg+135/1088]
unix_stream_recvmsg+0x87/0x440
Jul 12 22:20:41 mindpipe kernel: [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 12 22:20:41 mindpipe kernel: [sock_aio_read+155/176] sock_aio_read+0x9b/0xb0
Jul 12 22:20:41 mindpipe kernel: [do_sync_read+125/192] do_sync_read+0x7d/0xc0
Jul 12 22:20:41 mindpipe kernel: [do_select+435/752] do_select+0x1b3/0x2f0
Jul 12 22:20:41 mindpipe kernel: [vfs_read+223/256] vfs_read+0xdf/0x100
Jul 12 22:20:42 mindpipe kernel: [sys_read+45/80] sys_read+0x2d/0x50
Jul 12 22:20:42 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 12 22:20:42 mindpipe kernel:
Jul 12 22:20:44 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 12 22:20:44 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 12 22:20:44 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 12 22:20:44 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 12 22:20:44 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Oh dear, these don't make much sense.
You'll have the best chance of getting decent traces with
CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
will help a lot. Are you sure that CONFIG_PREEMPT is enabled on that
kernel?
19 Jul
19 Jul
9:02 a.m.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
Just as a reference point, what do you think is the
longest delay I
*should* be seeing? I recall hearing that BEOS guaranteed that
interrupts are never disabled for more than 50 usecs. This seems
achievable, as the average delay I am seeing is 15 usecs.
ATA hardirq latency can be as high as 700 usecs under load even on
modern hw, when big DMA requests are created with long scatter-gather
lists. We also moved some of the page IO completion code into irq
context which further increased hardirq latencies. Since these all touch
cold cachelines it all adds up quite quickly.
Ingo
9:52 a.m.
* Ingo Molnar <mingo(a)elte.hu> wrote:
ATA hardirq latency can be as high as 700 usecs under
load even on
modern hw, when big DMA requests are created with long scatter-gather
lists. We also moved some of the page IO completion code into irq
context which further increased hardirq latencies. Since these all
touch cold cachelines it all adds up quite quickly.
typically all of this happens with irqs enabled (unmask=1), but it still
increases scheduling latencies.
with the default DMA setup of today's ATA hw there can be a maximum of
256 entries in the sg-table all zapped in ide_end_request() ->
__end_that_request_first().
Plus the IDE driver also builds the sg-table of the next request in
hardirq context. (ide_build_dmatable() and ide_build_sglist()).
Ingo
20 Jul
20 Jul
3:52 a.m.
On Mon, 2004-07-19 at 06:48, Ingo Molnar wrote:
* Lee Revell <rlrevell(a)joe-job.com> wrote:
Does this scale in a linear fashion with respect to CPU speed? You
mentioned you were testing on a 2Ghz machine, does 700 usecs on that
translate to 2800 usecs on a 500Mhz box?
On a 2Ghz machine, 700 usecs is about one million CPU cycles. In other
words, the highest priority process can become runnable, then have to
wait *one million cycles* to get the CPU.
How much I/O do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt you
could improve this quite a bit. Disk throughput is good enough, anyone
in the real world who would feel a 10% hit would just throw hardware at
the problem.
Lee
Just as a reference point, what do you think is
the longest delay I
*should* be seeing? I recall hearing that BEOS guaranteed that
interrupts are never disabled for more than 50 usecs. This seems
achievable, as the average delay I am seeing is 15 usecs.
ATA hardirq latency can be as high as 700 usecs under load even on
modern hw, when big DMA requests are created with long scatter-gather
lists. We also moved some of the page IO completion code into irq
context which further increased hardirq latencies. Since these all touch
cold cachelines it all adds up quite quickly.
4:26 a.m.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
Does this scale in a linear fashion with respect to
CPU speed? You
mentioned you were testing on a 2Ghz machine, does 700 usecs on that
translate to 2800 usecs on a 500Mhz box?
given that this particular latency is dominated by cachemisses the DRAM
latency controls it too which is independent of CPU MHz. Wrt. cachemiss
costs, newer CPUs typically have twice the L2-cache line size (so it
takes more bus cycles to fetch it) - the improvements in bandwidth of
fetching a single line should offset most of this. DRAM latencies didnt
improve much in the past 10 years so that's almost a constant between a
500MHz/100MHz(system-bus) vs. 2GHz/400MHz system. So i'd guesstimate a
500 MHz box to do somewhere around 1000-1500 usecs.
How much I/O do you allow to be in flight at once? It
seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
limiting the in-flight IO will only work with IDE/PATA that doesnt have
multiple commands in flight for a given disk. SATA and SCSI handles
multiple command completions per IRQ invocation so limiting the size of
a single IO op has less effect there.
Ingo
5:13 a.m.
On Tue, 2004-07-20 at 02:12, Ingo Molnar wrote:
Yes, this would be a nice improvement.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
The particular system I am working with is a Via EPIA M-6000, which is
quite new but has a 600Mhz CPU due to it being fanless. I would imagine
this would put it closer to a 'modern system' in this regard, as it uses
DDR266.
Does this scale in a linear fashion with respect
to CPU speed? You
mentioned you were testing on a 2Ghz machine, does 700 usecs on that
translate to 2800 usecs on a 500Mhz box?
given that this particular latency is dominated by cachemisses the DRAM
latency controls it too which is independent of CPU MHz. Wrt. cachemiss
costs, newer CPUs typically have twice the L2-cache line size (so it
takes more bus cycles to fetch it) - the improvements in bandwidth of
fetching a single line should offset most of this. DRAM latencies didnt
improve much in the past 10 years so that's almost a constant between a
500MHz/100MHz(system-bus) vs. 2GHz/400MHz system. So i'd guesstimate a
500 MHz box to do somewhere around 1000-1500 usecs.
How much I/O
do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
limiting the in-flight IO will only work with IDE/PATA that doesnt have
multiple commands in flight for a given disk. SATA and SCSI handles
multiple command completions per IRQ invocation so limiting the size of
a single IO op has less effect there.
But the current behavior only causes latency problems for an IDE system,
so if this were made runtime-tunable then it would only be an issue for
SATA, right? This would cover 99.9% of audio users, who would gladly
trade some disk throughput for lower latency. You can record a *lot* of
tracks with even a few MB/s of disk throughput.
Lee
5:25 a.m.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
But the current behavior only causes latency problems
for an IDE
system, so if this were made runtime-tunable then it would only be an
issue for SATA, right? This would cover 99.9% of audio users, who
would gladly trade some disk throughput for lower latency. You can
record a *lot* of tracks with even a few MB/s of disk throughput.
it's an issue for all block IO drivers that do IO completions from IRQ
context and that can do DMA - i.e. every block IO hardware that uses
interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
tagged queueing active so there the latencies ought to be even higher
(although i havent measured this). IDE/PATA's limitation in this regard
limits latencies as well.
being able to control the max size of sg-tables and the max # of
outstanding commands per IRQ source [this later should already be
possible via driver options] should enable us to control the maximum
hardirq latency introduced by block IO.
(if the hardware & disk is fast enough, or you use a high # of
controllers and disks then you could still overload your system with a
stream of interrupts and cause unbound scheduling latencies - but this
is a separate problem.)
Ingo
5:30 p.m.
it's an issue for all block IO drivers that do IO
completions from IRQ
context and that can do DMA - i.e. every block IO hardware that uses
interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
I renew a question i asked earlier.
To my understanding, on a SMP or hyperthreading system, disabling of
IRQs is always local to one (virtual on HT) cpu.
So would it be possible to get ultralow latency by simply hardlock all irqs
and processes to cpu1 and the irq triggering the audiothread (together with
the audiothread) to cpu 2 using the sched_affinity and irq-affinity
capabilites of the kernel?
This would be an easy to use lowlatency hardware patch for linux audio users
with SMP/HT systems. Anybody knows?
I'm currently thinking about getting a new system and consider a dualsystem if
this worked.
Ralf
6:11 p.m.
On Tue, 2004-07-20 at 14:11, Ralf Beck wrote:
Should work. For example, the RTLinux people report excellent results
on SMP systems by binding all RT threads to one CPU and having the Linux
part of the system run on the other. This is just a "softer" version of
that setup. Even if there are cases where IRQs are disabled globally,
it would be an improvement. I suspect you are not getting much of a
response because no one has actually tested it with an audio system.
Lee
it's an
issue for all block IO drivers that do IO completions from IRQ
context and that can do DMA - i.e. every block IO hardware that uses
interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
I renew a question i asked earlier.
To my understanding, on a SMP or hyperthreading system, disabling of
IRQs is always local to one (virtual on HT) cpu.
So would it be possible to get ultralow latency by simply hardlock all irqs
and processes to cpu1 and the irq triggering the audiothread (together with
the audiothread) to cpu 2 using the sched_affinity and irq-affinity
capabilites of the kernel?
This would be an easy to use lowlatency hardware patch for linux audio users
with SMP/HT systems. Anybody knows?
I'm currently thinking about getting a new system and consider a dualsystem if
this worked.
10:02 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
While locking a RT process to a CPU to achieve even lower latencies
might be useful to some
the general userbase wants good latencies on simple UP, non HT-enabled
hardware too.
(AMD is gaining marketshare and we cannot simply expect that good
multimedia performance (aka low latency)
can be achieved only on those SMP/HT boxes where the truth is that
except in the case of really crappy hardware,
those common UP machines are actually capable to delives latencies in
the range of dozen of usecs.
(taking the RTLinux benchmarks as reference).
Of course Linux != RTLinux and we never expect nor pretend Linux to
match the response time of RTLinux but
as said earlier latencies at around 1msec should are doable on decent UP
boxes and this is more than
adequate to run demanding real time audio applications like software
synthesizers, samples, real time effects etc.
The only hope is that the hard work from Ingo, Andrew and all the others
gets soon integrated in the mainstream
2.6 kernel so all Linux users can take run demanding multimedia
applications out of the box without going through
the painful patching,kernel recompiling etc.
Or is this still not realistic at this point ?
(see the old issue with kernel 2.4, low latency patches were too
unclean to make it into the
ufficial kernel tree).
Plus what's very important is that every kernel developer and driver
developer (even thirdparty, especially those
that do closed source stuff like Nvidia etc) takes into account the
latency problems that code paths that run for
too long time (or disable IRQs for too long etc) might create.
While I'm not a kernel expert I assume the premptible kernel alleviates
this problem but I guess it still cannot
completely get rid of low latency-unfriendly routines and drivers.
Perhaps in upcoming Linux kernel/driver programming books this issue
should be stressed out and
mentioned prominently in order to avoid the eternal:
fixing latencies in kernel x.y.z -> new kernel versions with new
drivers/changes/enhancements comes out
-> latencies are bad again -> fix latencies in this new kernel version
..... ( iterate at infinitum)
cheers,
Benno
http://www.linuxsampler.org
Lee Revell wrote:
On Tue, 2004-07-20 at 14:11, Ralf Beck wrote:
Should work. For example, the RTLinux people report excellent results
on SMP systems by binding all RT threads to one CPU and having the Linux
part of the system run on the other. This is just a "softer" version of
that setup. Even if there are cases where IRQs are disabled globally,
it would be an improvement. I suspect you are not getting much of a
response because no one has actually tested it with an audio system.
Lee
it's
an issue for all block IO drivers that do IO completions from IRQ
context and that can do DMA - i.e. every block IO hardware that uses
interrupts. This includes SCSI too. In fact for SCSI it's a norm to have
I renew a question i asked earlier.
To my understanding, on a SMP or hyperthreading system, disabling of
IRQs is always local to one (virtual on HT) cpu.
So would it be possible to get ultralow latency by simply hardlock all irqs
and processes to cpu1 and the irq triggering the audiothread (together with
the audiothread) to cpu 2 using the sched_affinity and irq-affinity
capabilites of the kernel?
This would be an easy to use lowlatency hardware patch for linux audio users
with SMP/HT systems. Anybody knows?
I'm currently thinking about getting a new system and consider a dualsystem if
this worked.
10:47 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Tue, 2004-07-20 at 19:49, Benno Senoner wrote:
While locking a RT process to a CPU to achieve even
lower latencies
might be useful to some
the general userbase wants good latencies on simple UP, non HT-enabled
hardware too.
(AMD is gaining marketshare and we cannot simply expect that good
multimedia performance (aka low latency)
can be achieved only on those SMP/HT boxes where the truth is that
except in the case of really crappy hardware,
those common UP machines are actually capable to delives latencies in
the range of dozen of usecs.
(taking the RTLinux benchmarks as reference).
Of course Linux != RTLinux and we never expect nor pretend Linux to
match the response time of RTLinux but
as said earlier latencies at around 1msec should are doable on decent UP
boxes and this is more than
adequate to run demanding real time audio applications like software
synthesizers, samples, real time effects etc.
The only hope is that the hard work from Ingo, Andrew and all the others
gets soon integrated in the mainstream
2.6 kernel so all Linux users can take run demanding multimedia
applications out of the box without going through
the painful patching,kernel recompiling etc.
Or is this still not realistic at this point ?
(see the old issue with kernel 2.4, low latency patches were too
unclean to make it into the
ufficial kernel tree).
There are still a few areas that need work, ioctl gives me problems, but
the latest 2.6 kernels are quite good. If you look at the 'clean'
version of the voluntary kernel preemption patch it is pretty small. My
understanding is that the kernel is already preemptible anytime that a
spin lock (including the BKL) is not held, and that the voluntary kernel
preemption patch adds some scheduling points in places where it is safe
to sleep, but preemption is disabled because we are holding the BKL, and
that the number of these should approach zero as the kernel is improved
anyway.
Plus what's very important is that every kernel
developer and driver
developer (even thirdparty, especially those
that do closed source stuff like Nvidia etc) takes into account the
latency problems that code paths that run for
too long time (or disable IRQs for too long etc) might create.
While I'm not a kernel expert I assume the premptible kernel alleviates
this problem but I guess it still cannot
completely get rid of low latency-unfriendly routines and drivers.
Yes, this is important. One problem I had recently with the Via EPIA
board was that unless 2D acceleration was disabled by setting 'Option
"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
cause interrupts from the soundcard to be completely disabled for tens
of milliseconds. Users should keep in mind that by using 2D or 3D
hardware acceleration in X, you are allowing the X server to directly
access hardware, which can have very bad results if the driver is
buggy. I am not sure the kernel can do anything about this.
Lee
11:14 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Lee Revell wrote:
There are still a few areas that need work, ioctl
gives me problems, but
the latest 2.6 kernels are quite good. If you look at the 'clean'
version of the voluntary kernel preemption patch it is pretty small. My
understanding is that the kernel is already preemptible anytime that a
spin lock (including the BKL) is not held, and that the voluntary kernel
preemption patch adds some scheduling points in places where it is safe
to sleep, but preemption is disabled because we are holding the BKL, and
that the number of these should approach zero as the kernel is improved
anyway.
That's confusing to me. It was my understanding that the BKL is used to
completely lock down the kernel so that no other CPU can have a process
get into the kernel... something like how SMP was done under 2.0.
So, if you sleep during a BKL, wouldn't that imply that nothing else
would be allowed to enter the kernel until after the kernel thread that
took the lock wakes up and releases the lock?
21 Jul
21 Jul
9 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Tue, 20 Jul 2004 20:32:37 -0400
Lee Revell <rlrevell(a)joe-job.com> wrote:
Yes, this is important. One problem I had recently with the Via EPIA
board was that unless 2D acceleration was disabled by setting 'Option
"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
cause interrupts from the soundcard to be completely disabled for tens
of milliseconds. Users should keep in mind that by using 2D or 3D
hardware acceleration in X, you are allowing the X server to directly
access hardware, which can have very bad results if the driver is
buggy. I am not sure the kernel can do anything about this.
Hi,
interesting that you mention the Xserver. I use a dual graphics card setup atm [Nvidia GF3
TI and some matrox pci card]. The nvidia card seems to work flawlessly even with HW
accelleration [i use nvidias evil binary only drivers]. The matrox card OTH disturbs the
soundcard severely. Whenever i have activity on my second monitor i get sound artefacts in
jack's output [no cracklling, it's rather as if the volume is set to 0 for short
moments and then back to normal]. There's a certain chance that this artefact produces
an xrun. I suppose it's because the card is on the pci bus.
I figured it's maybe an irq issue problem, but whatever slot i put the gfx card in -
it made no difference [btw: how do i find out which resources this card uses? it is not
shown by /proc/interrupts]. I also tried putting the soundcard in many different slots to
maybe get it on higher prio irq, but it always gets irq 5 [according to
/proc/interrupts]..
Should i try a different 2nd gfx card? Should i avoid pci gfx cards at all costs? Will i
just have to live w/o second monitor? How do i find out which hw resources X is really
using?
Florian Schmidt
--
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/
9:34 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Florian Schmidt wrote :
Hi,
interesting that you mention the Xserver. I use a dual graphics card setup atm [Nvidia
GF3 TI and some matrox pci card]. The nvidia card seems to work flawlessly even with HW
accelleration [i use nvidias evil binary only drivers]. The matrox card OTH disturbs the
soundcard severely. Whenever i have activity on my second monitor i get sound artefacts in
jack's output [no cracklling, it's rather as if the volume is set to 0 for short
moments and then back to normal]. There's a certain chance that this artefact produces
an xrun. I suppose it's because the card is on the pci bus.
I figured it's maybe an irq issue problem, but whatever slot i put the gfx card in -
it made no difference [btw: how do i find out which resources this card uses? it is not
shown by /proc/interrupts]. I also tried putting the soundcard in many different slots to
maybe get it on higher prio irq, but it always gets irq 5 [according to
/proc/interrupts]..
Should i try a different 2nd gfx card? Should i avoid pci gfx cards at all costs? Will i
just have to live w/o second monitor? How do i find out which hw resources X is really
using?
Florian Schmidt
You could try to adjust the pci latency timer value of your graphic card
and sound card, see this link for a paper on the subject by Daniel
Robbins :
http://www-106.ibm.com/developerworks/library/l-hw2.html
Thomas
10:10 a.m.
New subject: pci gfx card / jack xruns / pci latencies (was: Re: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch)
On Wed, 21 Jul 2004 13:18:15 +0200
Thomas Charbonnel <thomas(a)undata.org> wrote:
You could try to adjust the pci latency timer value of
your graphic card
and sound card, see this link for a paper on the subject by Daniel
Robbins :
http://www-106.ibm.com/developerworks/library/l-hw2.html
Hi,
good idea. Sadly it doesn't make a difference. Interesting is that lspci -v
doesn't show any latency for my mga card:
tapas@mango:~$ lspci -v
[parts snipped, see end of mail]
0000:00:0d.0 Multimedia audio controller: Cirrus Logic CS 4614/22/24 [CrystalClear
SoundFusion Audio Accelerator] (rev 01)
Subsystem: TERRATEC Electronic GmbH: Unknown device 112e
Flags: bus master, medium devsel, latency 248, IRQ 5
Memory at cfffb000 (32-bit, non-prefetchable)
Memory at cfe00000 (32-bit, non-prefetchable) [size=1M]
Capabilities: <available only to root>
This is the relevant entry:
0000:00:0f.0 VGA compatible controller: Matrox Graphics, Inc. MGA 2064W [Millennium] (rev
01) (prog-if 00 [VGA])
Flags: stepping, medium devsel, IRQ 11
Memory at cfffc000 (32-bit, non-prefetchable) [size=cffe0000]
Memory at cd000000 (32-bit, prefetchable) [size=8M]
Expansion ROM at 00010000 [disabled]
0000:01:00.0 VGA compatible controller: nVidia Corporation NV20 [GeForce3 Ti 200] (rev a3)
(prog-if 00 [VGA])
Subsystem: Micro-Star International Co., Ltd.: Unknown device 8503
Flags: bus master, 66MHz, medium devsel, latency 248, IRQ 3
Memory at ce000000 (32-bit, non-prefetchable) [size=cfcf0000]
Memory at c0000000 (32-bit, prefetchable) [size=128M]
Memory at cca80000 (32-bit, prefetchable) [size=512K]
Expansion ROM at 00010000 [disabled]
Capabilities: <available only to root>
Besides not showing a latency the mga card seems to be the only device which has the Flag:
"stepping" instead of "bus master".. I tried setting the latency to 0
anyways, and increased it to ffh for my soundcard. The mga card has become really slow
though switching desktops still provokes xruns just as easily.. Any other mga card users
here? Is this normal behaviour for a dual gfx card setup? Other values didn't help
either..
The rest of the lspci -v output:
0000:00:00.0 Host bridge: Silicon Integrated Systems [SiS] 735 Host (rev 01)
Flags: bus master, medium devsel, latency 32
Memory at d0000000 (32-bit, non-prefetchable)
Capabilities: <available only to root>
0000:00:01.0 PCI bridge: Silicon Integrated Systems [SiS] Virtual PCI-to-PCI bridge (AGP)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 64
Bus: primary=00, secondary=01, subordinate=01, sec-latency=64
Memory behind bridge: cdc00000-cfcfffff
Prefetchable memory behind bridge: bc900000-ccafffff
0000:00:02.0 ISA bridge: Silicon Integrated Systems [SiS] SiS85C503/5513 (LPC Bridge)
Flags: bus master, medium devsel, latency 0
0000:00:02.1 SMBus: Silicon Integrated Systems [SiS]: Unknown device 0016
Flags: medium devsel
I/O ports at 0c00 [size=32]
0000:00:02.5 IDE interface: Silicon Integrated Systems [SiS] 5513 [IDE] (rev d0) (prog-if
80 [Master])
Subsystem: Silicon Integrated Systems [SiS] SiS5513 EIDE Controller (A,B step)
Flags: bus master, fast devsel, latency 128
I/O ports at ff00 [size=16]
0000:00:03.0 Ethernet controller: Silicon Integrated Systems [SiS] SiS900 PCI Fast
Ethernet (rev 90)
Subsystem: Elitegroup Computer Systems: Unknown device 0a14
Flags: bus master, medium devsel, latency 64, IRQ 10
I/O ports at dc00 [size=cffa0000]
Memory at cfff9000 (32-bit, non-prefetchable) [size=4K]
Expansion ROM at 00020000 [disabled]
Capabilities: <available only to root>
Thanks, Florian Schmidt
--
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/
10:40 a.m.
New subject: [linux-audio-dev] Re: pci gfx card / jack xruns / pci latencies
Hello Florian,
Is this with kernel 2.6.x or 2.4.x?
I remember there were some problems with Matrox cards, that were
initially not included in the 2.4.x low-latency patch, but I thought
that at some point Andrew included them.
http://www.eca.cx/lad/2002/06/0076.html
maarten
10:56 a.m.
New subject: [linux-audio-dev] Re: pci gfx card / jack xruns / pci latencies
On Wed, 21 Jul 2004 14:22:12 +0200
Maarten de Boer <mdeboer(a)iua.upf.es> wrote:
Hello Florian,
Is this with kernel 2.6.x or 2.4.x?
I remember there were some problems with Matrox cards, that were
initially not included in the 2.4.x low-latency patch, but I thought
that at some point Andrew included them.
http://www.eca.cx/lad/2002/06/0076.html
Hi,
thanks for your mail. It seems though that this is independent of the kernel version. I
booted into 2.4.22 with LL+preempt and it shows the exact same behaviour as did 2.6.7
vanilla, 2.6.7-bk10-ck5-H3 and 2.6.8-rc1-mm1... I think the card is just not good..
I'm also wondering about the weird lspci -v output which reports no latency for the
matrox card [and it's not bus master like all other pci devices i have]..
Flo
--
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/
11:26 a.m.
New subject: [linux-audio-dev] Re: pci gfx card / jack xruns / pci latencies
On Wed, 21 Jul 2004 14:49:46 +0200
Florian Schmidt <mista.tapas(a)gmx.net> wrote:
thanks for your mail. It seems though that this is
independent of the kernel version. I booted into 2.4.22 with LL+preempt and it shows the
exact same behaviour as did 2.6.7 vanilla, 2.6.7-bk10-ck5-H3 and 2.6.8-rc1-mm1... I think
the card is just not good.. I'm also wondering about the weird lspci -v output which
reports no latency for the matrox card [and it's not bus master like all other pci
devices i have]..
Btw: i use Xfree's "mga" driver. i never played with DRI, should i?
FLo
--
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/
12:50 p.m.
New subject: [linux-audio-dev] Re: pci gfx card / jack xruns / pci latencies
On Wed, Jul 21, 2004 at 02:49:46PM +0200, Florian Schmidt wrote:
On Wed, 21 Jul 2004 14:22:12 +0200
Maarten de Boer <mdeboer(a)iua.upf.es> wrote:
one other thing to try: be sure in you XF86Config you have
PciRetry set to off. From man mga:
Option "PciRetry" "boolean"
Enable or disable PCI retries. Default: off.
--
Paul Winkler
http://www.slinkp.com
Hello Florian,
Is this with kernel 2.6.x or 2.4.x?
I remember there were some problems with Matrox cards, that were
initially not included in the 2.4.x low-latency patch, but I thought
that at some point Andrew included them.
http://www.eca.cx/lad/2002/06/0076.html
Hi,
thanks for your mail. It seems though that this is independent of the kernel version. I
booted into 2.4.22 with LL+preempt and it shows the exact same behaviour as did 2.6.7
vanilla, 2.6.7-bk10-ck5-H3 and 2.6.8-rc1-mm1... I think the card is just not good..
I'm also wondering about the weird lspci -v output which reports no latency for the
matrox card [and it's not bus master like all other pci devices i have]..
1 p.m.
New subject: [linux-audio-dev] Re: pci gfx card / jack xruns / pci latencies
On Wed, 21 Jul 2004 10:35:18 -0400
Paul Winkler <pw_lists(a)slinkp.com> wrote:
one other thing to try: be sure in you XF86Config you
have
PciRetry set to off. From man mga:
Option "PciRetry" "boolean"
Enable or disable PCI retries. Default: off.
Thanks for the hint. I didn't even know man pages for the X11 drivers existed.. Cool.
Didn't help though ;) Well, anyways, i'll try some other pci cards today [as i
managed to talk a budy of mine into letting me have some for tests].. I'll let you
guys know my findings.
Florian Schmidt
--
Palimm Palimm!
Sounds/Ware:
http://affenbande.org/~tapas/
11:44 a.m.
New subject: [linux-audio-dev] Matrox (... Voluntary Kernel Preemption Patch)
On Wed, Jul 21, 2004 at 12:53:52PM +0200, Florian Schmidt wrote:
interesting that you mention the Xserver. I use a dual
graphics card
setup atm [Nvidia GF3 TI and some matrox pci card]. The nvidia card
seems to work flawlessly even with HW accelleration [i use nvidias
evil binary only drivers]. The matrox card OTH disturbs the soundcard
severely. Whenever i have activity on my second monitor i get sound
artefacts in jack's output [no cracklling, it's rather as if the
volume is set to 0 for short moments and then back to normal]. There's
a certain chance that this artefact produces an xrun. I suppose it's
because the card is on the pci bus.
i have had huge problems with AGP G400/G450 on more than one system.
i spent some time trying various driver and pci options, and although
cant swear that i systematically went through all the options,
i was unable to get them to work satisfactorily.
--
Tim Orford
4:04 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Wed, 2004-07-21 at 03:53, Florian Schmidt wrote:
On Tue, 20 Jul 2004 20:32:37 -0400
Lee Revell <rlrevell(a)joe-job.com> wrote:
The mga dri kernel driver shares a problem with the radeon (which I use
a lot) and the r128 drivers. They have high latency points that reach
10-15 msecs in normal operation[*]. I have a very old patch (not mine, I
don't quite remember where I got it from) that solves this, but it is
not a "legal" patch in the sense that in schedules with a lock held. It
seems to work but it will lock the computer at some point. AFAIK there
is no proper patch for this latency point at this time. Turning off
acceleration should get rid of the latency spikes with the usual
tradeoff of slow video performance.
-- Fernando
[*] with the (bad) patch:
http://ccrma.stanford.edu/~nando/latencytest/20040323/2.6.4-1.279.ll.ccrma.…
without the patch:
http://ccrma.stanford.edu/~nando/latencytest/20040323/2.6.4-1.279.ll.ccrma/
Yes, this is important. One problem I had
recently with the Via EPIA
board was that unless 2D acceleration was disabled by setting 'Option
"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
cause interrupts from the soundcard to be completely disabled for tens
of milliseconds. Users should keep in mind that by using 2D or 3D
hardware acceleration in X, you are allowing the X server to directly
access hardware, which can have very bad results if the driver is
buggy. I am not sure the kernel can do anything about this.
Hi,
interesting that you mention the Xserver. I use a dual graphics card setup atm
[Nvidia GF3 TI and some matrox pci card]. The nvidia card seems to work flawlessly
even with HW accelleration [i use nvidias evil binary only drivers]. The matrox
card OTH disturbs the soundcard severely. Whenever i have activity on my second
monitor i get sound artefacts in jack's output [no cracklling, it's rather as
if the volume is set to 0 for short moments and then back to normal]. There's
a certain chance that this artefact produces an xrun. I suppose it's because
the card is on the pci bus. 
22 Jul
22 Jul
8:40 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
--On Wednesday, 21 July 2004 12:53 p.m. +0200 Florian Schmidt
<mista.tapas(a)gmx.net> wrote:
Hi,
interesting that you mention the Xserver. I use a dual graphics card
setup atm [Nvidia GF3 TI and some matrox pci card]. The nvidia card seems
to work flawlessly even with HW accelleration [i use nvidias evil binary
only drivers]. The matrox card OTH disturbs the soundcard severely.
Whenever i have activity on my second monitor i get sound artefacts in
jack's output [no cracklling, it's rather as if the volume is set to 0
for short moments and then back to normal]. There's a certain chance that
this artefact produces an xrun. I suppose it's because the card is on the
pci bus.
<snip>
Should i try a different 2nd gfx card? Should i avoid
pci gfx cards at
all costs? Will i just have to live w/o second monitor? How do i find
out which hw resources X is really using?
Florian Schmidt
It is a PCI bus issue. You simply don't have enough PCI bus cycles
available to do what you want to do. The resource you're running out of is
bus bandwidth, and there's nothing to be done about it, other than remove
the PCI gfx card from the system.
If you get another dualhead AGP graphics card (anything will do), the
problem should go away. We have a developer who does lowlatency
multichannel sound stuff on a machine with a Matrox G450 dualhead card no
problem. I expect my own system (Radeon 9800 Pro and M-Audio 1010LT audio)
would be fine dualhead too, although I only run it singlehead at the
moment. The 1010LT is 10 channels in and out of 24-bit 96kHz audio and
works great down to 1.5ms buffers, so it is no small bus load itself.
Andrew
---------
Andrew McGregor
Director, Scientific Advisor
IndraNet Technologies Ltd
http://www.indranet-technologies.com/
-----BEGIN GEEK CODE BLOCK-----
Version: 3.1
GS/E/B/PA/SS d+(++) s+:+ a C++$ ULS++++ !P+++(---)$ L++++$ E++ W++ !N
w(+++) !O() M++ V--() Y+$ PGP+ t- !5? X- !R !tv@ b++(++++) DI++ D+++@ G
e+++ h(*)@ r%
------END GEEK CODE BLOCK------
9:22 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Fri, 23 Jul 2004 10:25:07 +1200
Andrew McGregor <andrew(a)indranet.co.nz> wrote:
It is a PCI bus issue. You simply don't have
enough PCI bus cycles
available to do what you want to do. The resource you're running out of is
bus bandwidth, and there's nothing to be done about it, other than remove
the PCI gfx card from the system.
If you get another dualhead AGP graphics card (anything will do), the
problem should go away. We have a developer who does lowlatency
multichannel sound stuff on a machine with a Matrox G450 dualhead card no
problem. I expect my own system (Radeon 9800 Pro and M-Audio 1010LT audio)
would be fine dualhead too, although I only run it singlehead at the
moment. The 1010LT is 10 channels in and out of 24-bit 96kHz audio and
works great down to 1.5ms buffers, so it is no small bus load itself.
Hi,
thanks for the advice. I have tried some more pci cards and a very slow Virge 64 or
something actually reduced the problem. Not completely though. I think i'll get me one
of those G450 dual head cards..
Florian Schmidt
--
Palimm Palimm!
http://affenbande.org/~tapas/
23 Jul
23 Jul
10:21 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Thu, 2004-07-22 at 19:16, Florian Schmidt wrote:
On Fri, 23 Jul 2004 10:25:07 +1200
Andrew McGregor <andrew(a)indranet.co.nz> wrote:
Be warned.. AFAIK, dual head on Matrox cards (At least the G400 series
anyway) requires the proprietary Matrox "HAL" library, which has caused
nothing but (severe) problems for everyone I've talked to who used it
(corrupt displays usually).
-DR-
It is a PCI bus issue. You simply don't have
enough PCI bus cycles
available to do what you want to do. The resource you're running out of is
bus bandwidth, and there's nothing to be done about it, other than remove
the PCI gfx card from the system.
If you get another dualhead AGP graphics card (anything will do), the
problem should go away. We have a developer who does lowlatency
multichannel sound stuff on a machine with a Matrox G450 dualhead card no
problem. I expect my own system (Radeon 9800 Pro and M-Audio 1010LT audio)
would be fine dualhead too, although I only run it singlehead at the
moment. The 1010LT is 10 channels in and out of 24-bit 96kHz audio and
works great down to 1.5ms buffers, so it is no small bus load itself.
Hi,
thanks for the advice. I have tried some more pci cards and a very slow Virge 64
or something actually reduced the problem. Not completely though. I think i'll
get me one of those G450 dual head cards..
Florian Schmidt 
24 Jul
24 Jul
2:20 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Fri, Jul 23, 2004 at 08:08:15PM -0400, Dave Robillard wrote:
Dual head on G450/G550 does not require the HAL unless you are using
DVI.
--
Ryan Underwood, <nemesis(a)icequake.net>
thanks for the advice. I have tried some more pci cards and a very slow Virge 64
or something actually reduced the problem. Not completely though. I think i'll
get me one of those G450 dual head cards..
Florian Schmidt
Be warned.. AFAIK, dual head on Matrox cards (At least the G400 series
anyway) requires the proprietary Matrox "HAL" library, which has caused
nothing but (severe) problems for everyone I've talked to who used it
(corrupt displays usually).
3:44 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Sat, 2004-07-24 at 00:05, Ryan Underwood wrote:
On Fri, Jul 23, 2004 at 08:08:15PM -0400, Dave
Robillard wrote:
Dual head on G450/G550 does not require the HAL unless you are using
DVI.
Ah, sorry; must be just the G400 then. My bad.
-DR-
thanks for the advice. I have tried some more pci cards and a very slow Virge 64
or something actually reduced the problem. Not completely though. I think i'll
get me one of those G450 dual head cards..
Florian Schmidt
Be warned.. AFAIK, dual head on Matrox cards (At least the G400 series
anyway) requires the proprietary Matrox "HAL" library, which has caused
nothing but (severe) problems for everyone I've talked to who used it
(corrupt displays usually).
5:45 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Sat, Jul 24, 2004 at 01:31:41PM -0400, Dave Robillard wrote:
Yeah, G400 DH and G400 MAX currently require it. I have some patches
in which are the beginnings of eliminating that.
--
Ryan Underwood, <nemesis(a)icequake.net>
Dual head on
G450/G550 does not require the HAL unless you are using
DVI.
Ah, sorry; must be just the G400 then. My bad.
21 Jul
21 Jul
12:40 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Lee Revell wrote:
anyway.
that's bad news.
VIA markets those mini-itx mainboards (with onboard audio/video/network)
as multimedia appliances and
therefore I'd expect the hardware providing low latencies when both
video acceleration and audio is used.
Hopefully only a driver issue (as in most of cases)
Since VIA released the unichrome (the gfx chipset contained in their
mainboards) sources someone should
contact these folks :
http://unichrome.sourceforge.net/
to check what is causing those latency spikes ?
Any unichrome developer lurking on LKML ?
cheers,
Benno
http://www.linuxsampler.org
Plus what's very important is that every
kernel developer and driver
developer (even thirdparty, especially those
that do closed source stuff like Nvidia etc) takes into account the
latency problems that code paths that run for
too long time (or disable IRQs for too long etc) might create.
While I'm not a kernel expert I assume the premptible kernel alleviates
this problem but I guess it still cannot
completely get rid of low latency-unfriendly routines and drivers.
Yes, this is important. One problem I had recently with the Via EPIA
board was that unless 2D acceleration was disabled by setting 'Option
"NoAccel"' in /etc/X11/XF86Config-4, overloading the X server would
cause interrupts from the soundcard to be completely disabled for tens
of milliseconds. Users should keep in mind that by using 2D or 3D
hardware acceleration in X, you are allowing the X server to directly
access hardware, which can have very bad results if the driver is
buggy. I am not sure the kernel can do anything about this.
Lee
20 Jul
20 Jul
12:33 p.m.
On Tue, Jul 20 2004, Ingo Molnar wrote:
elvtune is long dead :-)
it's not tweakable right now, but if you wish to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
+ blk_queue_max_sectors(drive->queue, 32);
+
printk("%s: max request size: %dKiB\n", drive->name,
drive->queue->max_sectors / 2);
/* Extract geometry if we did not already have one for the drive */
above will limit max request to 16kb, experiment as you see fit.
--
Jens Axboe
How much I/O
do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
7:46 p.m.
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
On Tue, Jul 20 2004, Ingo Molnar wrote:
elvtune is long dead :-)
it's not tweakable right now, but if you wish to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
+ blk_queue_max_sectors(drive->queue, 32);
+
printk("%s: max request size: %dKiB\n", drive->name,
drive->queue->max_sectors / 2);
/* Extract geometry if we did not already have one for the drive */
above will limit max request to 16kb, experiment as you see fit.
I will give this a try.
Using 2.6.8-rc1-mm1, with La Monte H. P. Yarroll's patch to daemonize
softirqs, running two jackds (one in the playback direction, one in
capture, to work around an issue with the emu10k1 driver) with 2 periods
of 32 frames (666 usec), I am no longer seeing any XRUNS at all once the
jackd processes are up and running. I tried stressing the filesystem by
recompiling ALSA while running a 'du /' and a 'find / -ls'. The
scheduling jitter peaks occasionally at around 100000 CPU cycles (166
usecs) every few seconds, but this is as high as it goes.
This is as good or better than a 2.4 kernel with the low latency
patches.
I do not have voluntary preemption enabled, as I do not have a version
of the patch that corresponds to this kernel.
The only XRUNS I still get seem related to ioctl() and mlockall().
These both cause an XRUN in one jackd process when starting another:
Jul 20 16:25:46 mindpipe kernel: ALSA
/home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 20 16:25:46 mindpipe kernel: [dump_stack+23/32] dump_stack+0x17/0x20
Jul 20 16:25:46 mindpipe kernel: [__crc_totalram_pages+115469/3518512]
snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
Jul 20 16:25:46 mindpipe kernel: [__crc_totalram_pages+291773/3518512]
snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
Jul 20 16:25:46 mindpipe kernel: [handle_IRQ_event+51/96] handle_IRQ_event+0x33/0x60
Jul 20 16:25:46 mindpipe kernel: [do_IRQ+161/320] do_IRQ+0xa1/0x140
Jul 20 16:25:46 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 20 16:25:46 mindpipe kernel: [do_anonymous_page+124/384]
do_anonymous_page+0x7c/0x180
Jul 20 16:25:46 mindpipe kernel: [do_no_page+78/784] do_no_page+0x4e/0x310
Jul 20 16:25:46 mindpipe kernel: [handle_mm_fault+193/368] handle_mm_fault+0xc1/0x170
Jul 20 16:25:46 mindpipe kernel: [get_user_pages+337/960] get_user_pages+0x151/0x3c0
Jul 20 16:25:46 mindpipe kernel: [make_pages_present+104/144]
make_pages_present+0x68/0x90
Jul 20 16:25:46 mindpipe kernel: [mlock_fixup+141/176] mlock_fixup+0x8d/0xb0
Jul 20 16:25:46 mindpipe kernel: [do_mlockall+112/144] do_mlockall+0x70/0x90
Jul 20 16:25:46 mindpipe kernel: [sys_mlockall+150/160] sys_mlockall+0x96/0xa0
Jul 20 16:25:46 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 20 16:25:46 mindpipe kernel: IRQ: delay = 9213566 cycles, jitter = 8820554
Jul 20 16:54:02 mindpipe kernel: ALSA
/home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D2c
Jul 20 16:54:02 mindpipe kernel: [dump_stack+23/32] dump_stack+0x17/0x20
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+115469/3518512]
snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+291159/3518512]
snd_emu10k1_interrupt+0xd1/0x3c0 [snd_emu10k1]
Jul 20 16:54:02 mindpipe kernel: [handle_IRQ_event+51/96] handle_IRQ_event+0x33/0x60
Jul 20 16:54:02 mindpipe kernel: [do_IRQ+161/320] do_IRQ+0xa1/0x140
Jul 20 16:54:02 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+86289/3518512]
snd_pcm_hw_rule_div+0x4b/0x60 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+77959/3518512]
snd_pcm_hw_refine+0x371/0x4a0 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+111312/3518512]
snd_pcm_hw_param_mask+0x3a/0x50 [snd_pcm]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+882570/3518512]
snd_pcm_oss_change_params+0xf4/0x850 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+884572/3518512]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+888499/3518512]
snd_pcm_oss_get_formats+0x1d/0x120 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+888796/3518512]
snd_pcm_oss_set_format+0x26/0x60 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel: [__crc_totalram_pages+894737/3518512]
snd_pcm_oss_ioctl+0x49b/0x740 [snd_pcm_oss]
Jul 20 16:54:02 mindpipe kernel: [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 20 16:54:02 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 20 16:54:02 mindpipe kernel: IRQ: delay = 6702030 cycles, jitter = 6302828
Lee
How much
I/O do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
24 Jul
24 Jul
2:22 a.m.
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
On Tue, Jul 20 2004, Ingo Molnar wrote:
elvtune is long dead :-)
it's not tweakable right now, but if you wish to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
+ blk_queue_max_sectors(drive->queue, 32);
+
printk("%s: max request size: %dKiB\n", drive->name,
drive->queue->max_sectors / 2);
/* Extract geometry if we did not already have one for the drive */
above will limit max request to 16kb, experiment as you see fit.
I tested this and the improvement is drastic. With the default value of
1024KB, running 'bonnie' produced XRUNS of 10+ ms. When this is changed
to 16KB, running bonnie only produces latency spikes of up to about 90
usecs, I did not see a single one hit 100usecs.
According to hdparm, the throughput is still quite good (42MB/sec on a
sub-$100 IDE drive).
This should definitely be made tunable, I would imagine this would be
easy to put in /proc. The default could stay at 1024KB, and users with
low latency requirements could lower it.
I am currently testing the effect on throughput and will have some
better numbers soon.
Lee
How much
I/O do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
4 a.m.
On Sat, 2004-07-24 at 00:07, Lee Revell wrote:
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
OK, here are my bonnie test results. They are not what I expected -
read and write performance is significantly better with 16KB max request
size, while seeking is much better with 1024KB.
jackd was running in the background in both cases. With 1024KB, there
were massive XRUNS, and worse, occasionally the soundcard interrupt was
completely lost for tens of milliseconds. This is what I would expect
if huge SG lists are being built in hardirq context. With 16KB, jackd
ran perfectly, the highest latency I was was about 100 usecs.
Kernel is 2.6.8-rc2 + voluntary-preempt-I4. CPU is 600Mhz, 512MB RAM.
16KB:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mindpipe 1G 3002 88 31164 60 13485 29 3035 84 32838 36 56.7 0
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 94 40 +++++ +++ 15510 66 204 86 +++++ +++ 906 83
mindpipe,1G,3002,88,31164,60,13485,29,3035,84,32838,36,56.7,0,16,94,40,+++++,+++,15510,66,204,86,+++++,+++,906,83
1024KB:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mindpipe 1G 2988 86 17647 29 10979 17 3114 90 28556 25 156.7 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 210 88 +++++ +++ 13295 62 215 89 +++++ +++ 920 84
mindpipe,1G,2988,86,17647,29,10979,17,3114,90,28556,25,156.7,1,16,210,88,+++++,+++,13295,62,215,89,+++++,+++,920,84
HD info:
/dev/hdc:
Model=Maxtor 6Y160P0, FwRev=YAR41BW0, SerialNo=Y44K8TZE
Config={ Fixed }
RawCHS=16383/16/63, TrkSize=0, SectSize=0, ECCbytes=57
BuffType=DualPortCache, BuffSize=7936kB, MaxMultSect=16, MultSect=16
CurCHS=4047/16/255, CurSects=16511760, LBA=yes, LBAsects=268435455
IORDY=on/off, tPIO={min:120,w/IORDY:120}, tDMA={min:120,rec:120}
PIO modes: pio0 pio1 pio2 pio3 pio4
DMA modes: mdma0 mdma1 mdma2
UDMA modes: udma0 udma1 udma2 udma3 udma4 udma5 *udma6
AdvancedPM=yes: disabled (255) WriteCache=enabled
Drive conforms to: (null):
* signifies the current active mode
Lee
On Tue, Jul 20 2004, Ingo Molnar wrote:
it's not tweakable right now, but if you wish
to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
+ blk_queue_max_sectors(drive->queue, 32);
+
I am currently testing the effect on throughput and will have some
better numbers soon. How much
I/O do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
4:48 a.m.
On Sat, 2004-07-24 at 01:46, Lee Revell wrote:
On Sat, 2004-07-24 at 00:07, Lee Revell wrote:
I repeated the test with 'bonnie -f' and without jackd running, and with
32KB vs. 1024KB. The results are not as drastic, which suggests that
all the overhead from the XRUN tracing was a factor.
These results show that 32KB is better than 1024KB in some areas, and
not singificantly worse in any of these metrics.
32KB:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mindpipe 1G 38499 68 16383 28 35913 34 154.5 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 257 99 +++++ +++ 24008 98 256 99 +++++ +++ 1179 98
mindpipe,1G,,,38499,68,16383,28,,,35913,34,154.5,1,16,257,99,+++++,+++,24008,98,256,99,+++++,+++,1179,98
1024KB:
Version 1.03 ------Sequential Output------ --Sequential Input- --Random-
-Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP /sec %CP
mindpipe 1G 38285 60 13961 20 33694 28 155.2 1
------Sequential Create------ --------Random Create--------
-Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
files /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP /sec %CP
16 259 99 +++++ +++ 24250 98 256 99 +++++ +++ 1184 98
mindpipe,1G,,,38285,60,13961,20,,,33694,28,155.2,1,16,259,99,+++++,+++,24250,98,256,99,+++++,+++,1184,98
Lee
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
OK, here are my bonnie test results. They are not what I expected -
read and write performance is significantly better with 16KB max request
size, while seeking is much better with 1024KB.
On Tue, Jul 20 2004, Ingo Molnar wrote:
I am currently testing the effect on throughput and will have some
better numbers soon. > How much I/O do you allow to be in flight at
once? It seems like by
> decreasing the maximum size of I/O that you handle in one interrupt
> you could improve this quite a bit. Disk throughput is good enough,
> anyone in the real world who would feel a 10% hit would just throw
> hardware at the problem.
it's not tweakable right now, but if you wish to experiment you just
need to add a line to ide-disk.c:idedisk_setup() - pseudo patch:
+ blk_queue_max_sectors(drive->queue, 32);
+
4:56 a.m.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
jackd was running in the background in both cases.
With 1024KB, there
were massive XRUNS, and worse, occasionally the soundcard interrupt
was completely lost for tens of milliseconds. This is what I would
expect if huge SG lists are being built in hardirq context. With
16KB, jackd ran perfectly, the highest latency I was was about 100
usecs.
Kernel is 2.6.8-rc2 + voluntary-preempt-I4. CPU is 600Mhz, 512MB RAM.
ok, i'll put in a tunable for the sg size.
Btw., it's not really the building of the SG list that is expensive,
it's the completion activity that is expensive since e.g. in the case of
ext3 IO traffic it goes over _every single_ sg entry with the following
fat codepath:
__end_that_request_first()
bio_endio()
end_bio_bh_io_sync()
journal_end_buffer_io_sync()
unlock_buffer()
wake_up_buffer()
bio_put()
bio_destructor()
mempool_free()
mempool_free_slab()
kmem_cache_free()
mempool_free()
mempool_free_slab()
kmem_cache_free()
the buffer_head, the bio and bio->bi_io_vec all lie on different
cachelines and are very likely to be not cached after long IO latencies.
So we eat at least 3 big cachemisses, times 256.
Jens, one solution would be to make BIO completion a softirq - like SCSI
does. That makes the latencies much easier to control.
Another thing would be to create a compound structure for bio and
[typical sizes of] bio->bi_io_vec and free them as one entity, this
would get rid of one of the cachemisses. (there cannot be a 3-way
compound structure that includes the bh too because the bh is freed
later on by ext3.)
Ingo
9:04 p.m.
On Sat, 2004-07-24 at 02:43, Ingo Molnar wrote:
* Lee Revell <rlrevell(a)joe-job.com> wrote:
I tested this with every power of two from 16KB to 1024KB. The current
default on my system is 1024KB. This may be affected by disk controller
or other factors, someone else reported a default of 128KB with the same
drive. Using jackd at the lowest reasonable latency setting (32
frames/period at 48000 frames/sec = 666 usecs/period ), the highest
value that does not cause problems is 256KB. The maximum latency spike
I saw with this setting was ~220 usecs.
For anyone unfamiliar, if jackd detects that it has been delayed by more
than half a period it considers this an error condition because even
though there was not an XRUN, it probably does not have enough time left
to process a block of audio, and even if it did, it would probably not
get scheduled in time to deliver it (IOW there would be an XRUN on the
next write) so it restarts.
I therefore propose 256KB as a good default for a low latency system
once this is made tunable. This might be a good default for any system,
I am not sure under what conditions this would be a bottleneck, but for
example this would let you handle multiple NFS clients using 8K or even
32K block sizes without compromising latency. For a single-user,
audio-centric system like a DAW, I would go as low as possible until you
see disk throughput start to drop, 16KB works well for me.
You would want this as high as possible for, say, a CD burner, but I
don't see the point is going as high as 1024KB for a disk drive.
Lee
jackd was running in the background in both
cases. With 1024KB, there
were massive XRUNS, and worse, occasionally the soundcard interrupt
was completely lost for tens of milliseconds. This is what I would
expect if huge SG lists are being built in hardirq context. With
16KB, jackd ran perfectly, the highest latency I was was about 100
usecs.
Kernel is 2.6.8-rc2 + voluntary-preempt-I4. CPU is 600Mhz, 512MB RAM.
ok, i'll put in a tunable for the sg size.
9:45 p.m.
On Sat, 2004-07-24 at 18:49, Lee Revell wrote:
On Sat, 2004-07-24 at 02:43, Ingo Molnar wrote:
Scratch this, I got a 380 usec spike with 256KB, enough to trigger jackd
to reatart. 128KB so far has only gone as high as 185 usecs.
Lee
ok, i'll put in a tunable for the sg size.
I therefore propose 256KB as a good default 
10:57 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Note that the method used by hdparm tends to underreport
achievable throughput somewhat, because it generally only
ever has one I/O "in flight".
Cheers
--
Mark Lord
(hdparm keeper & the original "Linux IDE Guy")
Lee Revell wrote:
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
>On Tue, Jul 20 2004, Ingo Molnar wrote:
>
>>>How much I/O do you allow to be in flight at once? It seems like by
>>>decreasing the maximum size of I/O that you handle in one interrupt
>>>you could improve this quite a bit. Disk throughput is good enough,
>>>anyone in the real world who would feel a 10% hit would just throw
>>>hardware at the problem.
...
According to hdparm, the throughput is still quite
good (42MB/sec on a
sub-$100 IDE drive).
3:13 a.m.
On Tue, 2004-07-20 at 08:19, Jens Axboe wrote:
On Tue, Jul 20 2004, Ingo Molnar wrote:
elvtune is long dead :-)
This one occured several times while bonnie was cleaning up after
itself:
ALSA /home/rlrevell/cvs/alsa/alsa-driver/alsa-kernel/core/pcm_lib.c:169:
XRUN: pcmC0D2c
[<c01066a7>] dump_stack+0x17/0x20
[<de935377>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
[<de945211>] snd_emu10k1_interrupt+0xd1/0x3c0 [snd_emu10k1]
[<c01078c3>] handle_IRQ_event+0x33/0x60
[<c0107c05>] do_IRQ+0xa5/0x170
[<c0106268>] common_interrupt+0x18/0x20
[<c0161b81>] shrink_dcache_parent+0x21/0x30
[<c0159d69>] d_unhash+0x39/0xb0
[<c0159e4c>] vfs_rmdir+0x6c/0x1b0
[<c015a05f>] sys_rmdir+0xcf/0xf0
[<c0106047>] syscall_call+0x7/0xb
ALSA /home/rlrevell/cvs/alsa/alsa-driver/alsa-kernel/core/pcm_lib.c:169:
XRUN: pcmC0D0p
[<c01066a7>] dump_stack+0x17/0x20
[<de935377>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
[<de945477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
[<c01078c3>] handle_IRQ_event+0x33/0x60
[<c0107c05>] do_IRQ+0xa5/0x170
[<c0106268>] common_interrupt+0x18/0x20
[<c0161b81>] shrink_dcache_parent+0x21/0x30
[<c0159d69>] d_unhash+0x39/0xb0
[<c0159e4c>] vfs_rmdir+0x6c/0x1b0
[<c015a05f>] sys_rmdir+0xcf/0xf0
[<c0106047>] syscall_call+0x7/0xb
Lee
How much
I/O do you allow to be in flight at once? It seems like by
decreasing the maximum size of I/O that you handle in one interrupt
you could improve this quite a bit. Disk throughput is good enough,
anyone in the real world who would feel a 10% hit would just throw
hardware at the problem.
i'm not sure whether this particular value (max # of sg-entries per IO
op) is runtime tunable. Jens? Might make sense to enable elvtune-alike
tunability of this value.
13 Jul
13 Jul
6:41 a.m.
On Mon, 2004-07-12 at 23:59, Andrew Morton wrote:
Oh dear, these don't make much sense.
You'll have the best chance of getting decent traces with
CONFIG_FRAME_POINTER=y and CONFIG_4KSTACKS=n, but I'm not sure that this
will help a lot. Are you sure that CONFIG_PREEMPT is enabled on that
kernel?
Here are some more. These happen when I switch from X to a text console
and back:
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+397/1888] do_con_write+0x18d/0x760
Jul 13 04:27:50 mindpipe kernel: [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel: [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel: [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel: [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [group_send_sig_info+101/144]
group_send_sig_info+0x65/0x90
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
Jul 13 04:27:50 mindpipe kernel: [scrup+242/320] scrup+0xf2/0x140
Jul 13 04:27:50 mindpipe kernel: [lf+96/112] lf+0x60/0x70
Jul 13 04:27:50 mindpipe kernel: [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 13 04:27:50 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
Jul 13 04:27:50 mindpipe kernel: [opost+162/464] opost+0xa2/0x1d0
Jul 13 04:27:50 mindpipe kernel: [write_chan+437/544] write_chan+0x1b5/0x220
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [__do_softirq+48/144] __do_softirq+0x30/0x90
Jul 13 04:27:50 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+1035/1888] do_con_write+0x40b/0x760
Jul 13 04:27:50 mindpipe kernel: [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel: [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel: [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel: [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+974/1888] do_con_write+0x3ce/0x760
Jul 13 04:27:50 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
Jul 13 04:27:50 mindpipe kernel: [opost+327/464] opost+0x147/0x1d0
Jul 13 04:27:50 mindpipe kernel: [write_chan+437/544] write_chan+0x1b5/0x220
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+1448/1888] do_con_write+0x5a8/0x760
Jul 13 04:27:50 mindpipe kernel: [con_write+27/48] con_write+0x1b/0x30
Jul 13 04:27:50 mindpipe kernel: [opost_block+210/400] opost_block+0xd2/0x190
Jul 13 04:27:50 mindpipe kernel: [__alloc_pages+458/864] __alloc_pages+0x1ca/0x360
Jul 13 04:27:50 mindpipe kernel: [write_chan+396/544] write_chan+0x18c/0x220
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [default_wake_function+0/32]
default_wake_function+0x0/0x20
Jul 13 04:27:50 mindpipe kernel: [tty_write+434/608] tty_write+0x1b2/0x260
Jul 13 04:27:50 mindpipe kernel: [write_chan+0/544] write_chan+0x0/0x220
Jul 13 04:27:50 mindpipe kernel: [vfs_write+186/256] vfs_write+0xba/0x100
Jul 13 04:27:50 mindpipe kernel: [sys_write+45/80] sys_write+0x2d/0x50
Jul 13 04:27:50 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:27:50 mindpipe kernel:
Lee
6:54 a.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [group_send_sig_info+101/144]
group_send_sig_info+0x65/0x90
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
Jul 13 04:27:50 mindpipe kernel: [scrup+242/320] scrup+0xf2/0x140
Jul 13 04:27:50 mindpipe kernel: [lf+96/112] lf+0x60/0x70
Jul 13 04:27:50 mindpipe kernel: [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 13 04:27:50 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
framebuffer scrolling inside lock_kernel(). Tricky. Suggest you use X or
vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.
7:03 a.m.
On Tue, 2004-07-13 at 04:43, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
OK, I figured this was not an easy one. I can just not do that.
Here are some more. These result from using mplayer with ALSA OSS
emulation:
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1394929/5353478]
snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2165815/5353478]
snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170556/5353478]
snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [kfree_skbmem+23/32] kfree_skbmem+0x17/0x20
Jul 13 04:31:49 mindpipe kernel: [__kfree_skb+101/240] __kfree_skb+0x65/0xf0
Jul 13 04:31:49 mindpipe kernel: [packet_rcv_spkt+380/544] packet_rcv_spkt+0x17c/0x220
Jul 13 04:31:49 mindpipe kernel: [dev_queue_xmit_nit+179/288]
dev_queue_xmit_nit+0xb3/0x120
Jul 13 04:31:49 mindpipe kernel: [avc_has_perm_noaudit+104/512]
avc_has_perm_noaudit+0x68/0x200
Jul 13 04:31:49 mindpipe kernel: [avc_has_perm+72/96] avc_has_perm+0x48/0x60
Jul 13 04:31:49 mindpipe kernel: [inode_has_perm+73/144] inode_has_perm+0x49/0x90
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1377866/5353478]
snd_pcm_hw_constraint_minmax+0x34/0x40 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1359876/5353478]
snd_pcm_hw_constraints_complete+0x17e/0x290 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [selinux_file_ioctl+213/784]
selinux_file_ioctl+0xd5/0x310
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2174928/5353478]
snd_pcm_oss_open+0x1aa/0x350 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [soundcore_open+328/768] soundcore_open+0x148/0x300
Jul 13 04:31:49 mindpipe kernel: [chrdev_open+233/528] chrdev_open+0xe9/0x210
Jul 13 04:31:49 mindpipe kernel: [dentry_open+260/560] dentry_open+0x104/0x230
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170812/5353478]
snd_pcm_oss_set_format+0x26/0x60 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2176960/5353478]
snd_pcm_oss_ioctl+0x51a/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 13 04:31:49 mindpipe kernel: [sys_fcntl64+87/144] sys_fcntl64+0x57/0x90
Jul 13 04:31:49 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1373142/5353478]
snd_interval_refine+0xa0/0x1c0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1357711/5353478]
snd_pcm_hw_rule_div+0x49/0x50 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1347991/5353478]
snd_pcm_hw_refine+0x371/0x4a0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1383137/5353478]
snd_pcm_hw_param_mask+0x3b/0x50 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2164618/5353478]
snd_pcm_oss_change_params+0xf4/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [chrdev_open+233/528] chrdev_open+0xe9/0x210
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170887/5353478]
snd_pcm_oss_get_format+0x11/0x30 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2176960/5353478]
snd_pcm_oss_ioctl+0x51a/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [sys_ioctl+256/608] sys_ioctl+0x100/0x260
Jul 13 04:31:49 mindpipe kernel: [sys_fcntl64+87/144] sys_fcntl64+0x57/0x90
Jul 13 04:31:49 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 04:31:49 mindpipe kernel:
Lee
Jul 13 04:27:50 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:27:50 mindpipe kernel: [group_send_sig_info+101/144]
group_send_sig_info+0x65/0x90
Jul 13 04:27:50 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:27:50 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:27:50 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:27:50 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:27:50 mindpipe kernel: [_mmx_memcpy+141/384] _mmx_memcpy+0x8d/0x180
Jul 13 04:27:50 mindpipe kernel: [scrup+242/320] scrup+0xf2/0x140
Jul 13 04:27:50 mindpipe kernel: [lf+96/112] lf+0x60/0x70
Jul 13 04:27:50 mindpipe kernel: [do_con_trol+2907/3360] do_con_trol+0xb5b/0xd20
Jul 13 04:27:50 mindpipe kernel: [do_con_write+1128/1888] do_con_write+0x468/0x760
Jul 13 04:27:50 mindpipe kernel: [con_put_char+51/64] con_put_char+0x33/0x40
framebuffer scrolling inside lock_kernel(). Tricky. Suggest you use X or
vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.
7:11 a.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Why not? You can certainly try removing those [un]lock_kernel() calls.
framebuffer scrolling inside lock_kernel().
Tricky. Suggest you use X or
vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.
OK, I figured this was not an easy one. I can just not do that. Here are some more. These result from using mplayer
with ALSA OSS
emulation:
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1394929/5353478]
snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2165815/5353478]
snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170556/5353478]
snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Looks like those memcpys in snd_pcm_format_set_silence() are hurting.
7:31 a.m.
At Tue, 13 Jul 2004 02:00:25 -0700,
Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Hmm, but it's not in lock at least...
Takashi
Here are some more. These result from using mplayer with ALSA OSS
emulation:
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1394929/5353478]
snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2165815/5353478]
snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170556/5353478]
snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Looks like those memcpys in snd_pcm_format_set_silence() are hurting.
7:36 a.m.
Takashi Iwai <tiwai(a)suse.de> wrote:
t Tue, 13 Jul 2004 02:00:25 -0700,
Andrew Morton wrote:
It is - see chrdev_open().
You can do unlock_kernel()/lock_kernel() in soundcore_open().
Lee Revell <rlrevell(a)joe-job.com> wrote:
Hmm, but it's not in lock at least...
Here are some more. These result from using mplayer with ALSA OSS
emulation:
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1394929/5353478]
snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2165815/5353478]
snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170556/5353478]
snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Looks like those memcpys in snd_pcm_format_set_silence() are hurting.
7:43 a.m.
At Tue, 13 Jul 2004 02:25:01 -0700,
Andrew Morton wrote:
Takashi Iwai <tiwai(a)suse.de> wrote:
Oh yeah...
t Tue, 13 Jul 2004 02:00:25 -0700,
Andrew Morton wrote:
It is - see chrdev_open().
Lee Revell <rlrevell(a)joe-job.com> wrote:
Hmm, but it's not in lock at least...
Here are some more. These result from using mplayer with ALSA OSS
emulation:
Jul 13 04:31:49 mindpipe kernel:
Jul 13 04:31:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 04:31:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 04:31:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 04:31:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+1394929/5353478]
snd_pcm_format_set_silence+0x4b/0x1d0 [snd_pcm]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2165815/5353478]
snd_pcm_oss_change_params+0x5a1/0x850 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2166620/5353478]
snd_pcm_oss_get_active_substream+0x76/0x90 [snd_pcm_oss]
Jul 13 04:31:49 mindpipe kernel: [__crc_totalram_pages+2170556/5353478]
snd_pcm_oss_get_formats+0x26/0x100 [snd_pcm_oss]
Looks like those memcpys in snd_pcm_format_set_silence() are hurting. You can do unlock_kernel()/lock_kernel() in
soundcore_open().
I remember ioctl() is also in lock_kernel()?
Takashi
7:55 a.m.
Takashi Iwai <tiwai(a)suse.de> wrote:
yes, you'll need to do unlock_kernel/lock_kernel there too. If someone
changes the rules, or otherwise calls your ioctl withoht lock_kernel() held
it will reliably go BUG with spinlock debugging enabled, so make sure you
test with CONFIG_DEBUG_SPINLOCK=y.
You can do unlock_kernel()/lock_kernel() in
soundcore_open().
I remember ioctl() is also in lock_kernel()?
7:33 a.m.
On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Why not? You can certainly try removing those [un]lock_kernel() calls.
Maybe I missed something. What exactly do you mean by 'make sure you're
wearing ear protection'?
Lee
framebuffer scrolling inside lock_kernel().
Tricky. Suggest you use X or
vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.
OK, I figured this was not an easy one. I can just not do that.
7:40 a.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
It might go boom. If it does screw up, it probably won't be very seriously
bad - maybe some display glitches. Just an experiment.
Lee Revell <rlrevell(a)joe-job.com> wrote:
Why not? You can certainly try removing those [un]lock_kernel() calls.
Maybe I missed something. What exactly do you mean by 'make sure you're
wearing ear protection'?
framebuffer scrolling inside lock_kernel().
Tricky. Suggest you use X or
vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
but make sure you're wearing ear protection.
OK, I figured this was not an easy one. I can just not do that.
6:47 p.m.
On Tue, 2004-07-13 at 05:29, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Heh, I thought you literally meant it could destroy my speakers. I have
a pretty loud amp hooked up to this thing! I will try that.
Meanwhile, here are some more XRUS traces from the same test, run
overnight, without any significant network activity. I believe updatedb
may have caused these. I left out the reiserfs related ones because
that seems to be a known issue.
Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:07 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:07 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:07 mindpipe kernel: [do_pollfd+125/144] do_pollfd+0x7d/0x90
Jul 13 06:22:07 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:07 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:07 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:07 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:07 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:07 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:07 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:07 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:07 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:07 mindpipe kernel: [__kmalloc+110/160] __kmalloc+0x6e/0xa0
Jul 13 06:22:07 mindpipe kernel: [sys_poll+209/560] sys_poll+0xd1/0x230
Jul 13 06:22:07 mindpipe kernel: [__pollwait+0/160] __pollwait+0x0/0xa0
Jul 13 06:22:07 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 06:22:07 mindpipe kernel:
Jul 13 06:22:19 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:22:19 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:22:19 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:22:19 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:22:19 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:22:19 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:22:19 mindpipe kernel:
Jul 13 06:26:13 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 06:26:13 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 06:26:13 mindpipe kernel: [__crc_totalram_pages+2260290/5353478]
find_attr+0xdc/0x250 [ntfs]
Jul 13 06:26:13 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 06:26:13 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 06:26:13 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 06:26:13 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 06:26:13 mindpipe kernel: [__crc_totalram_pages+2275930/5353478]
ntfs_readdir+0xd24/0xfe0 [ntfs]
Jul 13 06:26:13 mindpipe kernel: [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel: [vfs_readdir+133/160] vfs_readdir+0x85/0xa0
Jul 13 06:26:13 mindpipe kernel: [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel: [sys_getdents64+102/163] sys_getdents64+0x66/0xa3
Jul 13 06:26:13 mindpipe kernel: [filldir64+0/256] filldir64+0x0/0x100
Jul 13 06:26:13 mindpipe kernel: [syscall_call+7/11] syscall_call+0x7/0xb
Jul 13 06:26:13 mindpipe kernel:
Lee
On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
It might go boom. If it does screw up, it probably won't be very seriously
bad - maybe some display glitches. Just an experiment. Lee Revell <rlrevell(a)joe-job.com> wrote:
Maybe I missed something. What exactly do you mean by 'make sure you're
wearing ear protection'?
> framebuffer scrolling inside lock_kernel(). Tricky. Suggest you use X or
> vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
> but make sure you're wearing ear protection.
>
OK, I figured this was not an easy one. I can just not do that.
Why not? You can certainly try removing those [un]lock_kernel() calls.
8:42 p.m.
On Tue, 2004-07-13 at 05:29, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Seems to work perfectly. No visible display glitches.
I would imagine that putting a modem on a tty, then stressing the system
and watching for line errors would be a definitive test.
--- drivers/char/tty_io.c.orig 2004-07-13 16:55:28.000000000 -0400
+++ drivers/char/tty_io.c 2004-07-13 16:55:51.000000000 -0400
@@ -684,17 +684,13 @@
return -ERESTARTSYS;
}
if ( test_bit(TTY_NO_WRITE_SPLIT, &tty->flags) ) {
- lock_kernel();
written = write(tty, file, buf, count);
- unlock_kernel();
} else {
for (;;) {
unsigned long size = max((unsigned long)PAGE_SIZE*2, 16384UL);
if (size > count)
size = count;
- lock_kernel();
ret = write(tty, file, buf, size);
- unlock_kernel();
if (ret <= 0)
break;
written += ret;
Lee
On Tue, 2004-07-13 at 05:00, Andrew Morton wrote:
It might go boom. If it does screw up, it probably won't be very seriously
bad - maybe some display glitches. Just an experiment.
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
Lee Revell <rlrevell(a)joe-job.com> wrote:
Maybe I missed something. What exactly do you mean by 'make sure you're
wearing ear protection'?
> framebuffer scrolling inside lock_kernel(). Tricky. Suggest you use X or
> vgacon. You can try removing the lock_kernel() calls from do_tty_write(),
> but make sure you're wearing ear protection.
>
OK, I figured this was not an easy one. I can just not do that.
Why not? You can certainly try removing those [un]lock_kernel() calls.
6:17 a.m.
At Mon, 12 Jul 2004 17:46:39 -0700,
Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
It seems to be misplaced. A simple make-a-zillion-teeny-files test here
exhibits a 14 millisecond holdoff.
Yes, I measured a similar on reiserfs, too.
Creation and deletion of many small files causes high latencies.
The creation latency appears with ext3, too, but vanishes when
preemption is enabled.
The deletion peaks appear on all systems. It's in the long loop of
shrink_dcache_parent().
IIRC, the RCU tasklet stall was another cause of latency in some very
heavy cases.
But, except for creation/deletion of zillon-files, I've not seen any
big latencies during my tests with reiserfs. Reading/writing big
files (a few GB) are totally OK on reiserfs, all under 1.5 ms.
Takashi
resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
Interesting. There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work.
19 Jul
19 Jul
9:01 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
with the voluntary-preempt patch applied ext3 is below ~500 usecs for
all things i tried on a 2GHz CPU. Without the patch i can trigger
latencies up to milliseconds (even with CONFIG_PREEMPT) by triggering a
bigger commit stream via some large file write or a cached du / causing
a stream of atime updates. (I very much suspect that all other
journalled filesystems have similar problems and they'll need
measurements and fixing just like ext3 does.)
another bigger problem area is the VM - see my patch for details.
pagetable zapping and page reclaim are both problematic and need fixups
even under CONFIG_PREEMPT. Doing a simple 'make -j' kernel build that
hits swap triggers these easily. (after applying my patch the latencies
go below 1msec even with a 'make -j' overload.)
Ingo
Should I try
ext3?
ext3 is certainly better than that, but still has a couple of
potential problem spots. ext2 is probably the best at this time.
21 Jul
21 Jul
1:42 a.m.
On Mon, 2004-07-19 at 06:29, Ingo Molnar wrote:
* Andrew Morton <akpm(a)osdl.org> wrote:
with the voluntary-preempt patch applied ext3 is below ~500 usecs for
all things i tried on a 2GHz CPU. Without the patch i can trigger
latencies up to milliseconds (even with CONFIG_PREEMPT) by triggering a
bigger commit stream via some large file write or a cached du / causing
a stream of atime updates. (I very much suspect that all other
journalled filesystems have similar problems and they'll need
measurements and fixing just like ext3 does.)
another bigger problem area is the VM - see my patch for details.
pagetable zapping and page reclaim are both problematic and need fixups
even under CONFIG_PREEMPT. Doing a simple 'make -j' kernel build that
hits swap triggers these easily. (after applying my patch the latencies
go below 1msec even with a 'make -j' overload.)
I discovered I can reliably produce a large XRUN by toggling Caps Lock,
Scroll Lock, or Num Lock. This is with 2.6.8-rc1-mm1 + voluntary
preempt (I modified the patch by hand to apply on this kernel, as
2.6.8-rc2 disables my network card). Here is the XRUN trace.
ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
[<c01066f7>] dump_stack+0x17/0x20
[<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
[<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
[<c0107913>] handle_IRQ_event+0x33/0x60
[<c0107c55>] do_IRQ+0xa5/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c01d3f7f>] __delay+0xf/0x20
[<c021881a>] atkbd_sendbyte+0x5a/0xa0
[<c0218a35>] atkbd_command+0x1d5/0x200
[<c0218bcb>] atkbd_event+0x16b/0x200
[<c0215a95>] input_event+0x115/0x3d0
[<c01ecaeb>] kbd_bh+0xbb/0x160
[<c011a554>] tasklet_action+0x44/0x70
[<c011a303>] __do_softirq+0x83/0x90
[<c011a345>] do_softirq+0x35/0x40
[<c0107cc5>] do_IRQ+0x115/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c014b95e>] sys_read+0x2e/0x50
[<c0106097>] syscall_call+0x7/0xb
Lee
Should I
try ext3?
ext3 is certainly better than that, but still has a couple of
potential problem spots. ext2 is probably the best at this time.
2:19 a.m.
Lee Revell <rlrevell(a)joe-job.com> wrote:
discovered I can reliably produce a large XRUN by toggling Caps Lock,
Scroll Lock, or Num Lock. This is with 2.6.8-rc1-mm1 + voluntary
preempt
That's odd. I wonder if the hardware is sick. What is the duration is the
underrun? The info you sent didn't include that.
(I modified the patch by hand to apply on this kernel,
as
2.6.8-rc2 disables my network card).
eh? That's a rather more serious problem. Does the via-rhine.c from
2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?
ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
[<c01066f7>] dump_stack+0x17/0x20
[<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
[<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
[<c0107913>] handle_IRQ_event+0x33/0x60
[<c0107c55>] do_IRQ+0xa5/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c01d3f7f>] __delay+0xf/0x20
[<c021881a>] atkbd_sendbyte+0x5a/0xa0
[<c0218a35>] atkbd_command+0x1d5/0x200
[<c0218bcb>] atkbd_event+0x16b/0x200
[<c0215a95>] input_event+0x115/0x3d0
[<c01ecaeb>] kbd_bh+0xbb/0x160
[<c011a554>] tasklet_action+0x44/0x70
[<c011a303>] __do_softirq+0x83/0x90
[<c011a345>] do_softirq+0x35/0x40
[<c0107cc5>] do_IRQ+0x115/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c014b95e>] sys_read+0x2e/0x50
3:13 a.m.
On Wed, 2004-07-21 at 03:03, Andrew Morton wrote:
I posted this to LKML, the error I get is 'Invalid MAC Address'. I will
try the older version.
Lee
Lee Revell <rlrevell(a)joe-job.com> wrote:
I don't think so, it's a Dell QuietKey PS/2 keyboard, plugged into a
regular PS/2 port. The XRUN is 24-70ms, usually about 32ms.
discovered I can reliably produce a large XRUN by toggling Caps Lock,
Scroll Lock, or Num Lock. This is with 2.6.8-rc1-mm1 + voluntary
preempt
That's odd. I wonder if the hardware is sick. What is the duration is the
underrun? The info you sent didn't include that.
(I modified
the patch by hand to apply on this kernel, as
2.6.8-rc2 disables my network card).
eh? That's a rather more serious problem. Does the via-rhine.c from
2.6.8-rc1-mm1 work OK if you move it into 2.6.8-rc2?
ALSA /home/rlrevell/cvs/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
[<c01066f7>] dump_stack+0x17/0x20
[<de952477>] snd_pcm_period_elapsed+0x2c7/0x400 [snd_pcm]
[<de962477>] snd_emu10k1_interrupt+0x337/0x3c0 [snd_emu10k1]
[<c0107913>] handle_IRQ_event+0x33/0x60
[<c0107c55>] do_IRQ+0xa5/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c01d3f7f>] __delay+0xf/0x20
[<c021881a>] atkbd_sendbyte+0x5a/0xa0
[<c0218a35>] atkbd_command+0x1d5/0x200
[<c0218bcb>] atkbd_event+0x16b/0x200
[<c0215a95>] input_event+0x115/0x3d0
[<c01ecaeb>] kbd_bh+0xbb/0x160
[<c011a554>] tasklet_action+0x44/0x70
[<c011a303>] __do_softirq+0x83/0x90
[<c011a345>] do_softirq+0x35/0x40
[<c0107cc5>] do_IRQ+0x115/0x170
[<c01062b8>] common_interrupt+0x18/0x20
[<c014b95e>] sys_read+0x2e/0x50
3:53 a.m.
* Andrew Morton <akpm(a)osdl.org> wrote:
no, it's the ps2 driver that is sick. The ps2 keyboard driver is one of
the few places that busy-polls for IRQ completion from within a tasklet
context for things like led switching (yuck) - this can cause many
millisecs delays on all boards i tried.
Ingo
discovered I
can reliably produce a large XRUN by toggling Caps Lock,
Scroll Lock, or Num Lock. This is with 2.6.8-rc1-mm1 + voluntary
preempt
That's odd. I wonder if the hardware is sick. What is the duration is the
underrun? The info you sent didn't include that.
4:17 a.m.
On Wed, 2004-07-21 at 01:30, Ingo Molnar wrote:
* Andrew Morton <akpm(a)osdl.org> wrote:
no, it's the ps2 driver that is sick. The ps2 keyboard driver is one of
the few places that busy-polls for IRQ completion from within a tasklet
context for things like led switching (yuck) - this can cause many
millisecs delays on all boards i tried.
I can also add that La Monte H. P. Yarroll's patch to daemonize softirqs
seems to provide major improvements in latency (does not help this
problem of course). There has been at least one other patch posted to
LKML that that does the same thing. Will this feature be in the kernel
anytime soon?
Lee
discovered I can reliably produce a large XRUN by toggling Caps Lock,
Scroll Lock, or Num Lock. This is with 2.6.8-rc1-mm1 + voluntary
preempt
That's odd. I wonder if the hardware is sick. What is the duration is the
underrun? The info you sent didn't include that.
8:05 a.m.
* Lee Revell <rlrevell(a)joe-job.com> wrote:
I can also add that La Monte H. P. Yarroll's patch
to daemonize
softirqs seems to provide major improvements in latency (does not help
this problem of course). There has been at least one other patch
posted to LKML that that does the same thing. Will this feature be in
the kernel anytime soon?
we already 'daemonize' softirqs in the stock kernel if the load is high
enough. (this is what ksoftirqd does) So the only question is a tunable
to make this deferring of softirq load mandatory. Yarroll's patch is
quite complex, i dont think that is necessary. It also has at least one
conceptual problem, the NOP-ing of local_bh_disable/enable in case of
CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?
I've added a very simple solution to daemonize softirqs similar to
Yarroll's patch to the -H5 version of voluntary-preempt:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H5
the 'level' of preemption is controlled via the value of
voluntary_preempt - a level of 2 (default) means that softirq-deferring
is done too. A level of 1 means only the extra preemption points are
activated, level 0 means the vanilla scheduling and softirq behavior.
below i've also attached a softirq.c patch against 2.6.8-rc2 that does
unconditional deferring. (this patch is of course not intended to be
merged upstream as-is, since normally we want to process softirqs right
after the irq context.)
Ingo
--- linux/kernel/softirq.c.orig
+++ linux/kernel/softirq.c
@@ -70,7 +70,7 @@ static inline void wakeup_softirqd(void)
*/
#define MAX_SOFTIRQ_RESTART 10
-asmlinkage void __do_softirq(void)
+asmlinkage void ___do_softirq(void)
{
struct softirq_action *h;
__u32 pending;
@@ -106,6 +106,23 @@ restart:
__local_bh_enable();
}
+asmlinkage void __do_softirq(void)
+{
+ /*
+ * 'preempt harder'. Push all softirq processing off to ksoftirqd.
+ */
+ if (local_softirq_pending())
+ wakeup_softirqd();
+}
+
+asmlinkage void _do_softirq(void)
+{
+ local_irq_disable();
+ ___do_softirq();
+ local_irq_enable();
+}
+
+
#ifndef __ARCH_HAS_DO_SOFTIRQ
asmlinkage void do_softirq(void)
@@ -340,7 +363,7 @@ static int ksoftirqd(void * __bind_cpu)
preempt_disable();
if (cpu_is_offline((long)__bind_cpu))
goto wait_to_die;
- do_softirq();
+ _do_softirq();
preempt_enable();
cond_resched();
}
9:28 a.m.
below i've also attached a softirq.c patch against
2.6.8-rc2 that does
unconditional deferring. (this patch is of course not intended to be
merged upstream as-is, since normally we want to process softirqs
right after the irq context.)
i've got a more complete patch against vanilla 2.6.8-rc2:
http://redhat.com/~mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2
which introduces the following tunable:
/proc/sys/kernel/defer_softirqs (default: 0)
this, if enabled, causes all softirqs to be processed within ksoftirqd,
and it also breaks out of the softirq loop if preemption of ksoftirqd
has been triggered by a higher-prio task.
I've also added this additional break-out to the -H6 patch of
voluntary-preempt:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H6
it's enabled by default.
Ingo
9:47 a.m.
Ingo Molnar wrote:
What do you think about deferring softirqs just while in critical
sections?
I'm not sure how well this works, and it is CONFIG_PREEMPT only
but in theory it should prevent unbounded softirqs while under
locks without taking the performance hit of doing the context
switch.
below i've
also attached a softirq.c patch against 2.6.8-rc2 that does
unconditional deferring. (this patch is of course not intended to be
merged upstream as-is, since normally we want to process softirqs
right after the irq context.)
i've got a more complete patch against vanilla 2.6.8-rc2:
http://redhat.com/~mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2
which introduces the following tunable:
/proc/sys/kernel/defer_softirqs (default: 0)
this, if enabled, causes all softirqs to be processed within ksoftirqd,
and it also breaks out of the softirq loop if preemption of ksoftirqd
has been triggered by a higher-prio task.
I've also added this additional break-out to the -H6 patch of
voluntary-preempt:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H6
it's enabled by default.
4:52 p.m.
On Wed, Jul 21, 2004 at 09:32:46PM +1000, Nick Piggin wrote:
What do you think about deferring softirqs just while
in critical
sections?
I'm not sure how well this works, and it is CONFIG_PREEMPT only
but in theory it should prevent unbounded softirqs while under
locks without taking the performance hit of doing the context
switch.
You're still running do_softirq() with preemption disabled, which is
almost as bad as doing it under a lock.
-Scott
5:31 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
You're still running do_softirq() with preemption
disabled, which is
almost as bad as doing it under a lock.
well softirqs are designed like that. I've added extra preemption code
to the latest patch to avoid repeat processing.
Ingo
6:11 p.m.
On Wed, Jul 21, 2004 at 08:46:50PM +0200, Ingo Molnar wrote:
* Scott Wood <scott(a)timesys.com> wrote:
And those who wish to continue using them like that can. However, in
my patch they never run with preemption disabled, which can result in
substantial latency improvement (as long as nothing else is causing
similar delays). I see nothing in the design that *requires* them to
continue running with preemption disabled.
Likewise, interrupts are "designed" to be unpreemptible, but it is
possible to run them in their own threads so as to further reduce
sources of latency (at a throughput cost, of course). This allows
long-held spinlocks that an interrupt handler needs to acquire to be
replaced with mutexes that don't inhibit preemption.
Of course, a better fix is to keep the interrupt handlers and
critical sections short, but threading them can be very effective for
producing low latencies in the short term (we were able to achieve
worst measured case latencies of well under 100us on ordinary PC
hardware under 2.4.x using this approach).
You're still running do_softirq() with
preemption disabled, which is
almost as bad as doing it under a lock.
well softirqs are designed like that. I've added extra preemption code to the latest
patch to avoid
repeat processing.
Do you also add preemption checks in all of the various loops that
can be run from within a single softirq instance?
-Scott
10 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
Likewise, interrupts are "designed" to be
unpreemptible, but it is
possible to run them in their own threads so as to further reduce
sources of latency (at a throughput cost, of course). This allows
long-held spinlocks that an interrupt handler needs to acquire to be
replaced with mutexes that don't inhibit preemption.
Of course, a better fix is to keep the interrupt handlers and critical
sections short, but threading them can be very effective for producing
low latencies in the short term (we were able to achieve worst
measured case latencies of well under 100us on ordinary PC hardware
under 2.4.x using this approach).
do you have a 2.6 patch for hardirq redirection too? I always thought
this to be the best approach to achive hard-RT class latency guarantees
under Linux (but never coded it up). The problem with RTLinux is that it
introduces a separate OS (with separate APIs). It is (much) further
ahead of Linux in terms of latencies, algorithms and guarantees but is
still a separate OS. I believe there is a natural synergy between low
latencies needed for good desktop and multiuser performance and
soft-RT/hard-RT needs, which we should use - RTLinux doesnt generate
this synergy.
if both hardirqs and softirqs are redirectable to process contexts then
the only unpredictable latency would be the very short IRQ entry stub of
a new hardirq costing ~5 usecs - which latency is limited in effect
unless the CPU is hopelessly bombarded with interrupts.
to solve the spinlock problem of hardirqs i'd propose a dual type
spinlock that is a spinlock if hardirqs are immediate (synchronous) and
it would be a mutex if hardirqs are redirected (asynchronous). Then some
simple driver could be converted to this RT-aware spinlock and we'd see
how well it works. Have you done experiments in this direction? I
believe this all could be merged upstream, given sufficient cleanliness.
Ingo
22 Jul
22 Jul
12:43 a.m.
On Wed, Jul 21, 2004 at 11:45:34PM +0200, Ingo Molnar wrote:
do you have a 2.6 patch for hardirq redirection too?
Yes, and I'll try to get it cleaned up and posted soon. It depends
on threaded softirqs, though (so I'll have to do something with the
local_bh_disable; probably put it back to the way it was, at least
for now).
I always thought this to be the best approach to
achive hard-RT
class latency guarantees under Linux (but never coded it up). The
problem with RTLinux is that it introduces a separate OS (with
separate APIs).
If you really want hard-RT-type guarantees, though, you'd need to go
with a scheme whereby interrupts/preemption are disabled only where
the code has specifically been "approved" as not capable of causing
excessive latency (which IRQ threads are a step towards, and which is
what we did in 2.4), as opposed to the current scheme of benchmarking
latencies and fixing those places which actually show up as being
problematic. There's always the possibility that a different usage
case will show up later, causing the latency in some piece of code to
go way up.
if both hardirqs and softirqs are redirectable to
process contexts then
the only unpredictable latency would be the very short IRQ entry stub of
a new hardirq costing ~5 usecs - which latency is limited in effect
unless the CPU is hopelessly bombarded with interrupts.
Those aren't the only sources; you also have preempt_disable() and
such (as well as hardware weirdness, though there's not much we can
do about that).
to solve the spinlock problem of hardirqs i'd
propose a dual type
spinlock that is a spinlock if hardirqs are immediate (synchronous) and
it would be a mutex if hardirqs are redirected (asynchronous). Then some
simple driver could be converted to this RT-aware spinlock and we'd see
how well it works. Have you done experiments in this direction?
This sort of substitution is what we did in 2.4, though we made this
type the default and gave the real spinlocks a new name to be used in
those few places where it was really needed. Of course, this
resulted in a lot of places using a mutex where a spinlock would have
been fine.
-Scott
5:54 a.m.
* Scott Wood <scott(a)timesys.com> wrote:
i did not say latency sources, i said unpredictable latency sources.
hardirq and softirq processing introduces near arbitrary latency at any
(irqs-enabled) point in the kernel. Hence they make all kernel code
unbound in essence - even the most trivial kernel code, sys_getpid().
by (optionally) moving softirqs and hardirqs to a process context we've
removed this source of uncertainty by making all processing synchronous,
and what remains are ordinary algorithmic properties of the kernel code
- which we can predict and control. I.e. with those two fixed we now
have a _chance_ to guarantee latencies.
[the only remaining source of 'latency uncertainty' is the small
asynchronous hardirq stub that would still remain. This has an effect
that can be compared to e.g. cache effects and it cannot become unbound
unless the CPU is bombarded with a very high number of interrupts.]
what are those few places where a spinlock was really needed?
I'm a bit uneasy about making mutexes the default not due to performance
but due to e.g. some hardware being very timing-sensitive. I dont think
we can make a blanket assumption that every code path is preemptable if
all processing is synchronous and the exclusions are listened to.
Ingo
if both
hardirqs and softirqs are redirectable to process contexts then
the only unpredictable latency would be the very short IRQ entry stub of
a new hardirq costing ~5 usecs - which latency is limited in effect
unless the CPU is hopelessly bombarded with interrupts.
Those aren't the only sources; you also have preempt_disable() and
such (as well as hardware weirdness, though there's not much we can do
about that). to solve the
spinlock problem of hardirqs i'd propose a dual type
spinlock that is a spinlock if hardirqs are immediate (synchronous) and
it would be a mutex if hardirqs are redirected (asynchronous). Then some
simple driver could be converted to this RT-aware spinlock and we'd see
how well it works. Have you done experiments in this direction?
This sort of substitution is what we did in 2.4, though we made this
type the default and gave the real spinlocks a new name to be used in
those few places where it was really needed. Of course, this resulted
in a lot of places using a mutex where a spinlock would have been
fine.
5:08 p.m.
On Thu, Jul 22, 2004 at 09:40:34AM +0200, Ingo Molnar wrote:
* Scott Wood <scott(a)timesys.com> wrote:
Places that inherently cannot sleep, such as inside the scheduler,
the unthreadable part of the hard IRQ code, inside the mutex
implementation, etc.
This sort of substitution is what we did in 2.4,
though we made this
type the default and gave the real spinlocks a new name to be used in
those few places where it was really needed. Of course, this resulted
in a lot of places using a mutex where a spinlock would have been
fine.
what are those few places where a spinlock was really needed? I'm a bit uneasy about making mutexes the default
not due to performance
but due to e.g. some hardware being very timing-sensitive.
In practice, this didn't turn out to be an issue; most modern
hardware seems to be pretty tolerant of that (and you already have to
deal with things like interrupts getting in the way), and drivers
which do local_irq_disable() or to ensure timing will still work.
Has this sort of problem been seen with RT-Linux and such, which
would cause similar delays?
-Scott
6 p.m.
On Thu, Jul 22, 2004 at 02:53:08PM -0400, Scott Wood wrote:
On Thu, Jul 22, 2004 at 09:40:34AM +0200, Ingo Molnar
wrote:
Scott and Ingo,
Drivers that you might have to poll (technically a kind of hardware level
spinlock multi-device concurrency problem... Those special spins need to be
bounded by a preempt_{dis,en}able to prevent deadlocking) for completion since
it can't do an async notify of some sort, the low level timer handling
infrastructure, all places in the scheduler, all child functions/places that
can sleep/block within a non-preemptable critical section must be demoted back
to spinlocks and more stuff. The problem isn't trivial at all. The latter one
on the list is something that requires progressive audit as minor kernel
releases come out.
This is major project. kgdb needs to be ported to it, etc... and other
things that slip my mind right now.
There are also other problem with moving to a largely sleeping mutex
style kernel, dead lock detection becomes sorely needed. Current spinlock
detection methods are probably going to be useless in a system like this.
BSD/OS-FreeBSD have some of these facilites. The TimeSys mutexes have a
read/write lock, akind to BSD/OS shared/exclusive locks, that has a depth
first search cycle detector in them. My background is BSD/OS-FreeBSD.
The problem with FreeBSD is that their project is pretty much crap
in that they have these ego-manics focused on cute little SMP mechanisms
instead of actually doing the hard work of pushing locks down into lower
level subsystems. That's why DragonFly-BSD split from them in addition
to other multipule reasons. They also have tons more class and are nicer,
smarter folks in general. :)
The good thing about this, Linux, is that many of the RTOS issues we're
talking about now have a direct 1:1 relationship to SMP locking issues.
The concurrency issues, in a mathematical manner, are the same. It's not
just the trivial stuff with holding a non-preemptive lock such as BLK, but
in the context of priority inheritance and contention where, I believe,
the simple case use of priority inheritance should be considered a kind
contention overload condition of the overall system.
The general purpose OS folks like FreeBSD and old versions Solaris have a
sort of confused and backward notion of this problem by using priority
inheritance as a substitute for fixing the contention problems in the
first place. They abuse the scheduler to try and a get system that can
respond regularly to priority centric schedulers, but what happens instead
is they spread scheduling irregularities that effect the entire kernel
in unknown, unpredictable and odd ways. RTOS folks are much more sensitive
to the over and inappropriate use of things like this.
Linux is prime for some kind of RTOS conversion in that locks have been
pushed down sufficiently that late/runtime detection logic with simple
priority inheritance would rarely be triggered. That's basically a kind
of lock contention maladaptation. There's a possibility of making Linux
a top-notch RTOS if the right folks where working on that stuff.
Rant over... :)
BTW, this basically turning Linux into a kind of SGI IRIX style system.
* Scott Wood <scott(a)timesys.com> wrote:
Places that inherently cannot sleep, such as inside the scheduler,
the unthreadable part of the hard IRQ code, inside the mutex
implementation, etc. This sort of substitution is what we did in 2.4,
though we made this
type the default and gave the real spinlocks a new name to be used in
those few places where it was really needed. Of course, this resulted
in a lot of places using a mutex where a spinlock would have been
fine.
what are those few places where a spinlock was really needed? I'm a bit
uneasy about making mutexes the default not due to performance
but due to e.g. some hardware being very timing-sensitive.
In practice, this didn't turn out to be an issue; most modern
hardware seems to be pretty tolerant of that (and you already have to
deal with things like interrupts getting in the way), and drivers
which do local_irq_disable() or to ensure timing will still work. Has this sort of problem been seen with RT-Linux and
such, which
would cause similar delays?
bill
28 Jul
28 Jul
7:16 a.m.
* Bill Huey <bhuey(a)lnxw.com> wrote:
There are also other problem with moving to a largely
sleeping mutex
style kernel, dead lock detection becomes sorely needed. Current
spinlock detection methods are probably going to be useless in a
system like this. [...]
i have another worry with the 'everything is a mutex' concept. Currently
we still do have a number of 'central' locks such as dcache_lock, or the
SLAB locks. So even if all (but the scheduling) spinlocks are converted
to sleeping mutexes what do you gain? A high-prio RT task will get to
execute userspace instructions almost immediately, but any kernel
functionality use of this RT thread might still be blocked by a priority
inversion problem. So the same type of latency problems that we are
detecting and solving currently will occur on a mutex-based system as
well - if the RT application wants to use kernel functionality.
so why dont we keep the spinlocks [on UP, nonpreemptible sections] and
just let the kernel finish its work and get to a fully lock-quiescent
state ASAP where we can reschedule? It's not like that a high-prio RT
task can avoid this situation with any guarantee, as long as these
central locks remain. (in fact it would have to do this processing with
higher overhead, because the lowprio thread that got preemption needs to
be boosted, scheduled, unscheduled and the high-prio task needs to
schedule again.)
i'd agree with turning most of the finegrained per-task (non-irq-safe)
spinlocks into mutexes (or spin-mutexes). But the central locks that an
RT task would likely hit need to remain spinlocks i believe.
plus there are central mutexes too that are in 'hiding' currently but
could cause latencies just as much.
Ingo
7:07 p.m.
On Wed, Jul 28, 2004 at 08:45:47AM +0200, Ingo Molnar wrote:
i have another worry with the 'everything is a
mutex' concept. Currently
we still do have a number of 'central' locks such as dcache_lock, or the
SLAB locks. So even if all (but the scheduling) spinlocks are converted
to sleeping mutexes what do you gain? A high-prio RT task will get to
execute userspace instructions almost immediately, but any kernel
functionality use of this RT thread might still be blocked by a priority
inversion problem. So the same type of latency problems that we are
detecting and solving currently will occur on a mutex-based system as
well - if the RT application wants to use kernel functionality.
The difference is that latency issues are isolated to code which uses
the locks with bad critical sections. If some random driver holds an
internal lock for a few milliseconds, it won't slow down anything
that doesn't interact with the driver (either directly or by blocking
on something else that interacts with the driver). Turning the
spinlocks into mutexes doesn't eliminate the need for lock-breaking;
it just lets the lock-breaking effort be concentrated on the core
mutexes, so you don't need to fix *all* of the bad critical sections
to see a reduction in worst case latency of common code.
Those critical sections where lock-breaking has been done can be
converted back into spinlocks. Essentially, mutexes would be used
for "untrusted" locks, as opposed to using spinlocks just where
they're absolutely necessary. Over time, the set of trusted locks
would presumably go up, though we'd have to be careful to make sure
people know that they need to be especially careful of latency issues
when they touch code that uses such locks.
One of the main benefits is that it significantly increases the RT
guarantees for those users for whom the RT portion of their app can
be verified as only using a limited, testable/auditable subset of
kernel paths. Otherwise, you run the risk of some special code path
(such as out-of-memory, perhaps) that didn't get hit in latency
testing showing up and causing problems for a RT task that doesn't
need memory, and doesn't go anywhere near the lock in question. How
much this matters depends on how hard the RT one needs is.
Another is that the observed latencies of many apps is likely to go
down faster than when squashing each individual latency in the system
as it shows up. Said squashing can (and should) still occur, but at
least some apps will not have to wait.
-Scott
29 Jul
29 Jul
4:35 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
Those critical sections where lock-breaking has been
done can be
converted back into spinlocks. Essentially, mutexes would be used for
"untrusted" locks, as opposed to using spinlocks just where they're
absolutely necessary. Over time, the set of trusted locks would
presumably go up, though we'd have to be careful to make sure people
know that they need to be especially careful of latency issues when
they touch code that uses such locks.
One of the main benefits is that it significantly increases the RT
guarantees for those users for whom the RT portion of their app can be
verified as only using a limited, testable/auditable subset of kernel
paths. [...]
ok, i see - this makes 100% sense. I'm wondering how intrusive such an
all-preemptive patchset is? There are some problems with per-CPU data
structures on SMP. Right now holding a spinlock means one can use
smp_processor_id() and rely on it staying constant in the critical
section. With a mutex in the same place all such assumptions would
break. Is there some automatic way to deal with these issues (or to at
least detect them reliably?).
Ingo
4:51 p.m.
On Thu, Jul 29, 2004 at 08:21:10PM +0200, Ingo Molnar wrote:
* Scott Wood <scott(a)timesys.com> wrote:
ok, i see - this makes 100% sense. I'm wondering how intrusive such an
all-preemptive patchset is? There are some problems with per-CPU data
structures on SMP. Right now holding a spinlock means one can use
smp_processor_id() and rely on it staying constant in the critical
section. With a mutex in the same place all such assumptions would
break. Is there some automatic way to deal with these issues (or to at
least detect them reliably?).
Make smp_processor_id check if preempt_count() is non-zero to make sure
that you're running within a non-preemptable critical section (scheduler
deferred). Do the same with local_irq_* critical section by checking to
see if interrupts are disabled. They are also non-preemptable (hardware
defered).
That's my suggestion.
bill
5:33 p.m.
On Thu, Jul 29, 2004 at 11:36:26AM -0700, Bill Huey wrote:
On Thu, Jul 29, 2004 at 08:21:10PM +0200, Ingo Molnar
wrote:
There are legitimate reasons to use smp_processor_id() outside of a
non-preemptible region, though. These include debugging
printks/logging, and situations where the awareness of CPU is for
optimization rather than correctness (for example, using per-cpu data
with per-cpu locks, which are only contended if preemption occurs, or
per-cpu counters incremented with atomic operations (or where counter
accuracy is not critical)).
The detection mechanism we used in 2.4 was simply to grep for
smp_processor_id and check/fix everything manually (which is somewhat
tedious, but there aren't *too* many instances in core code, and many
uses are similar to one another).
-Scott
* Scott Wood <scott(a)timesys.com> wrote:
ok, i see - this makes 100% sense. I'm wondering how intrusive such an
all-preemptive patchset is? There are some problems with per-CPU data
structures on SMP. Right now holding a spinlock means one can use
smp_processor_id() and rely on it staying constant in the critical
section. With a mutex in the same place all such assumptions would
break. Is there some automatic way to deal with these issues (or to at
least detect them reliably?).
Make smp_processor_id check if preempt_count() is non-zero to make sure
that you're running within a non-preemptable critical section (scheduler
deferred). Do the same with local_irq_* critical section by checking to
see if interrupts are disabled. They are also non-preemptable (hardware
defered).
5:40 p.m.
On Thu, Jul 29, 2004 at 03:17:52PM -0400, Scott Wood wrote:
On Thu, Jul 29, 2004 at 11:36:26AM -0700, Bill Huey
wrote:
There are legitimate reasons to use smp_processor_id() outside of a
non-preemptible region, though. These include debugging
printks/logging, and situations where the awareness of CPU is for
optimization rather than correctness (for example, using per-cpu data
with per-cpu locks, which are only contended if preemption occurs, or
per-cpu counters incremented with atomic operations (or where counter
accuracy is not critical)).
The detection mechanism we used in 2.4 was simply to grep for
smp_processor_id and check/fix everything manually (which is somewhat
tedious, but there aren't *too* many instances in core code, and many
uses are similar to one another).
That's a better method. But if there's a need for a kind of runtime
detector, I guess you could do that. The use of smp_processor_id() should
be seldom enough that manually fixing all of the points really should
be the solution.
bill
30 Jul
30 Jul
12:38 a.m.
On Wed, 2004-07-28 at 02:45, Ingo Molnar wrote:
i'd agree with turning most of the finegrained
per-task (non-irq-safe)
spinlocks into mutexes (or spin-mutexes). But the central locks that an
RT task would likely hit need to remain spinlocks i believe.
plus there are central mutexes too that are in 'hiding' currently but
could cause latencies just as much.
Here are patches that convert spinlocks into kernel mutexes with
priority inheritance. They look reasonably clean, and might be
interesting to try.
http://inf3-www.informatik.unibw-muenchen.de/research/linux/mutex/mutex.html
Best regards,
Eric St-Laurent
29 Jul
29 Jul
6:55 p.m.
Hi!
i did not say latency sources, i said unpredictable
latency sources.
hardirq and softirq processing introduces near arbitrary latency at any
(irqs-enabled) point in the kernel. Hence they make all kernel code
unbound in essence - even the most trivial kernel code, sys_getpid().
...
[the only remaining source of 'latency
uncertainty' is the small
asynchronous hardirq stub that would still remain. This has an effect
that can be compared to e.g. cache effects and it cannot become unbound
unless the CPU is bombarded with a very high number of interrupts.]
Well, I do not follow you I guess.
With large-enough number of hardirqs you do no progress at all.
Even if only "sane" number of irqs, if they all decide to hit within one
getpid(), this getpid is going to take quite long....
Pavel
--
64 bytes from 195.113.31.123: icmp_seq=28 ttl=51 time=448769.1 ms
7:14 p.m.
On Thu, Jul 29, 2004 at 10:26:30PM +0200, Pavel Machek wrote:
Well, I do not follow you I guess.
With large-enough number of hardirqs you do no progress at all.
Even if only "sane" number of irqs, if they all decide to hit within one
getpid(), this getpid is going to take quite long....
Pavel
Ordinarily, yes. However, if it's a high-priority RT task that does
the getpid(), whose priority is higher than that of the RT tasks,
you'll get at most one hardirq stub per active IRQ number; after
that, the IRQs will be masked until their threads get a chance to be
scheduled.
-Scott
7:16 p.m.
Hi!
But will not even num_IRQs*time_per_stub be so high that any analysis
is impractical?
...
...
Hmm, that high-priority hask only has to eat num_IRQs*time_per_stub
once, so perhaps its okay.
Pavel
--
People were complaining that M$ turns users into beta-testers...
...jr ghea gurz vagb qrirybcref, naq gurl frrz gb yvxr vg gung jnl!
Well, I do not
follow you I guess.
With large-enough number of hardirqs you do no progress at all.
Even if only "sane" number of irqs, if they all decide to hit within one
getpid(), this getpid is going to take quite long....
Pavel
Ordinarily, yes. However, if it's a high-priority RT task that does
the getpid(), whose priority is higher than that of the RT tasks,
you'll get at most one hardirq stub per active IRQ number; after
that, the IRQs will be masked until their threads get a chance to be
scheduled.
7:24 p.m.
* Pavel Machek <pavel(a)ucw.cz> wrote:
...
yes, all of this assumes some _minimal_ sanity of the hardware
environment. We do detect interrupt storms and turn those IRQ sources
off, but there's no (sane) way to avoid interrupt storms from
driver-handled IRQ sources.
Ingo
[the only remaining source of 'latency
uncertainty' is the small
asynchronous hardirq stub that would still remain. This has an effect
that can be compared to e.g. cache effects and it cannot become unbound
unless the CPU is bombarded with a very high number of interrupts.]
Well, I do not follow you I guess.
With large-enough number of hardirqs you do no progress at all.
Even if only "sane" number of irqs, if they all decide to hit within
one getpid(), this getpid is going to take quite long....
22 Jul
22 Jul
12:04 a.m.
* Scott Wood <scott(a)timesys.com> wrote:
And those who wish to continue using them like that can. However, in
my patch they never run with preemption disabled, which can result in
substantial latency improvement (as long as nothing else is causing
similar delays). I see nothing in the design that *requires* them to
continue running with preemption disabled.
did you get my other mail in which i explained how e.g. the networking
code _relies_ on the softirq semantics?
of course you can improve latencies by making something preemptible that
wasnt preemptible before, but if you do that you should be absolutely
sure that it can be done without breaking that code. And in this case it
cannot be done. The comments in your patch also suggest that you were
unsure about this issue:
+/* As far as I can tell, local_bh_disable() didn't stop ksoftirqd
+ from running before. Since all softirqs now run from one of
+ the ksoftirqds, this shouldn't be necessary. */
+
+static inline void local_bh_disable(void)
+{
+}
local_bh_disable() of course stops ksoftirqd from running on that CPU.
You're still running do_softirq() with preemption
disabled, which is
almost as bad as doing it under a lock.
well softirqs are designed like that. Do you also add preemption checks in all of the
various loops that can
be run from within a single softirq instance?
no, that's the next step, if those loops turn out to be problematic.
E.g. network device backlogs default to a value of 100 or so which
shouldnt generate too bad latencies. If it does it's easy to break out
of those loops, they are already breakout-aware.
Ingo
1:02 a.m.
On Wed, Jul 21, 2004 at 11:08:32PM +0200, Ingo Molnar wrote:
I suppose it depends on what sort of latencies you're trying to
achieve, and how fast the hardware you're running on is. In our 2.4
kernel we typically achieved worst case latencies of 50us or so on PC
hardware, and a few hundred us on slower embedded hardware. Packet
processing would have to be awfully fast to cram 100 packets in that
time...
-Scott
did you get my other mail in which i explained how
e.g. the networking
code _relies_ on the softirq semantics?
Yes, but not until after I sent that e-mail.
Do you also
add preemption checks in all of the various loops that can
be run from within a single softirq instance?
no, that's the next step, if those loops turn out to be problematic.
E.g. network device backlogs default to a value of 100 or so which
shouldnt generate too bad latencies. If it does it's easy to break out
of those loops, they are already breakout-aware.
21 Jul
21 Jul
9:07 p.m.
Scott Wood wrote:
On Wed, Jul 21, 2004 at 09:32:46PM +1000, Nick Piggin
wrote:
do_softirq is only run a maximum of once with preemption disabled,
instead of an unbounded number of times.
What do you think about deferring softirqs just
while in critical
sections?
I'm not sure how well this works, and it is CONFIG_PREEMPT only
but in theory it should prevent unbounded softirqs while under
locks without taking the performance hit of doing the context
switch.
You're still running do_softirq() with preemption disabled, which is
almost as bad as doing it under a lock.
22 Jul
22 Jul
12:18 a.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
What do you think about deferring softirqs just while
in critical
sections?
I'm not sure how well this works, and it is CONFIG_PREEMPT only but in
theory it should prevent unbounded softirqs while under locks without
taking the performance hit of doing the context switch.
i dont think this is sufficient. A high-prio RT task might be performing
something that is important to it but isnt in any critical section. This
includes userspace processing. We dont want to delay it with softirqs.
Ingo
3:11 a.m.
Ingo Molnar wrote:
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
Given that we're looking for something acceptable for 2.6, how about
adding
if (rt_task(current))
kick ksoftirqd instead
Otherwise, what is the performance penalty of doing all softirq
processing from ksoftirqd?
What do you think about deferring softirqs just
while in critical
sections?
I'm not sure how well this works, and it is CONFIG_PREEMPT only but in
theory it should prevent unbounded softirqs while under locks without
taking the performance hit of doing the context switch.
i dont think this is sufficient. A high-prio RT task might be performing
something that is important to it but isnt in any critical section. This
includes userspace processing. We dont want to delay it with softirqs.
5:21 a.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
Given that we're looking for something acceptable
for 2.6, how about
adding
if (rt_task(current))
kick ksoftirqd instead
Otherwise, what is the performance penalty of doing all softirq
processing from ksoftirqd?
this is insufficient too. An RT task might be _waiting to run_ and
spending our time in a non-RT context (including the idle task) doing
softirq processing might delay it indefinitely.
what we could do is to add a rq->nr_running_rt and do the deferred
softirq processing unconditionally if (rq->nr_running_rt). I'd still add
a sysctl to make it unconditional for user processes too - if someone
really cares about latency and doesnt want to make all his tasks RT.
I'll code this up for the next version of the patch.
Ingo
9:09 a.m.
Ingo Molnar wrote:
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
Or just if (rt_task || need_resched). Another thing that may be
worthwhile thinking about is allowing softirqs to be run directly
from interrupt if the idle thread is running, maybe with an early
exit if something becomes runnable. Although this all may be going
down the path of too much complexity.
Maybe just the simple approach of "if you care about hard latency
then let ksoftirq do everything" is best?
Given that we're looking for something
acceptable for 2.6, how about
adding
if (rt_task(current))
kick ksoftirqd instead
Otherwise, what is the performance penalty of doing all softirq
processing from ksoftirqd?
this is insufficient too. An RT task might be _waiting to run_ and
spending our time in a non-RT context (including the idle task) doing
softirq processing might delay it indefinitely.
what we could do is to add a rq->nr_running_rt and do the deferred
softirq processing unconditionally if (rq->nr_running_rt). I'd still add
a sysctl to make it unconditional for user processes too - if someone
really cares about latency and doesnt want to make all his tasks RT.
I'll code this up for the next version of the patch.
2:37 p.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
this doesnt work either: once we've committed ourselves to do an
'immediate' softirq processing pass we are risking latencies. We cannot
preempt the idle task while it's processing softirqs the same way we can
do the lock-break if they are always deferred.
Also, it's not really a performance issue if the system is idle then can
switch to ksoftirqd cheaply (lazy-TLB) and once we switch away from
ksoftirqd it costs as much as switching away from the idle task.
Ingo
this is
insufficient too. An RT task might be _waiting to run_ and
spending our time in a non-RT context (including the idle task) doing
softirq processing might delay it indefinitely.
what we could do is to add a rq->nr_running_rt and do the deferred
softirq processing unconditionally if (rq->nr_running_rt). I'd still add
a sysctl to make it unconditional for user processes too - if someone
really cares about latency and doesnt want to make all his tasks RT.
I'll code this up for the next version of the patch.
Or just if (rt_task || need_resched). Another thing that may be
worthwhile thinking about is allowing softirqs to be run directly from
interrupt if the idle thread is running, maybe with an early exit if
something becomes runnable. Although this all may be going down the
path of too much complexity.
8:26 p.m.
Ingo Molnar wrote:
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
this doesnt work either: once we've committed ourselves to do an
'immediate' softirq processing pass we are risking latencies. We cannot
preempt the idle task while it's processing softirqs the same way we can
do the lock-break if they are always deferred.
It is a preempt off region no matter where it is run. I don't see how
moving it to ksoftirqd can shorten that time any further.
this is
insufficient too. An RT task might be _waiting to run_ and
spending our time in a non-RT context (including the idle task) doing
softirq processing might delay it indefinitely.
what we could do is to add a rq->nr_running_rt and do the deferred
softirq processing unconditionally if (rq->nr_running_rt). I'd still add
a sysctl to make it unconditional for user processes too - if someone
really cares about latency and doesnt want to make all his tasks RT.
I'll code this up for the next version of the patch.
Or just if (rt_task || need_resched). Another thing that may be
worthwhile thinking about is allowing softirqs to be run directly from
interrupt if the idle thread is running, maybe with an early exit if
something becomes runnable. Although this all may be going down the
path of too much complexity. Also, it's not really a performance issue if the
system is idle then can
switch to ksoftirqd cheaply (lazy-TLB) and once we switch away from
ksoftirqd it costs as much as switching away from the idle task.
Yeah you are probably right.
23 Jul
23 Jul
4:02 a.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
look at my latest patches to see how it's done. We can preempt softirq
handlers via lock-break methods. The same method doesnt work in the idle
thread. With this method i've reduced worst-case softirq latencies from
~2-4 msecs to 100-200 usecs on a 2GHz x86 box.
Ingo
this doesnt
work either: once we've committed ourselves to do an
'immediate' softirq processing pass we are risking latencies. We cannot
preempt the idle task while it's processing softirqs the same way we can
do the lock-break if they are always deferred.
It is a preempt off region no matter where it is run. I don't see how
moving it to ksoftirqd can shorten that time any further.
5 a.m.
Ingo Molnar wrote:
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
look at my latest patches to see how it's done. We can preempt softirq
handlers via lock-break methods. The same method doesnt work in the idle
Are you referring to this patch?
http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2
Surely something similar could easily be done for irq context softirq
processing with a patch like my earlier one? And it would prevent spilling
to ksoftirq when a RT thread isn't waiting to run.
this
doesnt work either: once we've committed ourselves to do an
'immediate' softirq processing pass we are risking latencies. We cannot
preempt the idle task while it's processing softirqs the same way we can
do the lock-break if they are always deferred.
It is a preempt off region no matter where it is run. I don't see how
moving it to ksoftirqd can shorten that time any further. thread. With this method i've reduced worst-case
softirq latencies from
~2-4 msecs to 100-200 usecs on a 2GHz x86 box.
Nice numbers.
5:08 a.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
no, i'm referring to this one:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-I1
(disregard the debugging induced complexity.)
look at my
latest patches to see how it's done. We can preempt softirq
handlers via lock-break methods. The same method doesnt work in the idle
Are you referring to this patch?
http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2 Surely something similar could easily be done for irq
context softirq
processing with a patch like my earlier one? And it would prevent
spilling to ksoftirq when a RT thread isn't waiting to run.
the softirq-defer patch is just the first step to enable lock-break of
softirqs - the lock-break is done in the -I1 patch.
(what patch do you refer to via 'my earlier one'? Did you mean your 'if
(rt_task())' suggestion?)
Ingo
5:26 a.m.
Ingo Molnar wrote:
Well yeah, obviously it would need a bit of help before it gets there.
But I'll shut up for now because I'm not the one doing all the coding :)
It seems like you're getting good results.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
no, i'm referring to this one:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-I1
(disregard the debugging induced complexity.)
OK... help me out here, you are referring to this:
+/*
+ * Preempt a softirq context if necessary:
+ */
+int cond_resched_softirq(void)
+{
+ BUG_ON(!in_softirq());
+ if (softirq_need_resched()) {
+ __local_bh_enable();
+ __cond_resched();
+ local_bh_disable();
+ return 1;
+ }
+ return 0;
+}
To break out of softirq processing, right?
You wouldn't need to do this to break out of interrupt context softirqs
because you wouldn't bother returning to it. Just hand the work off to
ksoftirqd.
The main thing I am looking at is getting low latency softirqs without
always handing them off to ksoftirqd. Not to mention that my patch
disables softirqs in critical regions, which can't be bad for scalability.
look at my
latest patches to see how it's done. We can preempt softirq
handlers via lock-break methods. The same method doesnt work in the idle
Are you referring to this patch?
http://people.redhat.com/mingo/voluntary-preempt/defer-softirqs-2.6.8-rc2-A2 Surely something similar could easily be done for
irq context softirq
processing with a patch like my earlier one? And it would prevent
spilling to ksoftirq when a RT thread isn't waiting to run.
the softirq-defer patch is just the first step to enable lock-break of
softirqs - the lock-break is done in the -I1 patch.
(what patch do you refer to via 'my earlier one'? Did you mean your 'if
(rt_task())' suggestion?)
5:35 a.m.
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
You wouldn't need to do this to break out of
interrupt context
softirqs because you wouldn't bother returning to it. Just hand the
work off to ksoftirqd.
this is plainly not the case. Look at eg. the net_tx_action() lock-break
i did in the -I1 patch. There we first create a private queue which we
work down. With my approach we can freely reschedule _within the loop_.
With your suggestion this is not possible.
i.e. executing a softirq in a process context gives us all the
advantages of a process context: all the local state is saved and
preserved until the preemption is done. These advantages are not there
for either immediate or idle-task-only-immediate type of softirq
processing.
Ingo
5:43 a.m.
Ingo Molnar wrote:
* Nick Piggin <nickpiggin(a)yahoo.com.au> wrote:
Sorry I missed that. Yeah if you are seeing high latencies *within*
a single softirq then my thing obviously wouldn't work.
If they're as high as a couple of ms on your 2GHz machine, then they
definitely shouldn't be processed in the interrupt path, so yeah
doing them in process context is the best thing to do.
You wouldn't need to do this to break out of
interrupt context
softirqs because you wouldn't bother returning to it. Just hand the
work off to ksoftirqd.
this is plainly not the case. Look at eg. the net_tx_action() lock-break
i did in the -I1 patch. There we first create a private queue which we
work down. With my approach we can freely reschedule _within the loop_.
With your suggestion this is not possible.
21 Jul
21 Jul
4:47 p.m.
On Wed, Jul 21, 2004 at 10:52:46AM +0200, Ingo Molnar wrote:
this, if enabled, causes all softirqs to be processed
within ksoftirqd,
and it also breaks out of the softirq loop if preemption of ksoftirqd
has been triggered by a higher-prio task.
You'll still have the latency of finishing the currently executing
softirq, which often includes a loop itself (whose break condition is
based on not hogging the CPU, rather than letting higher priority
tasks in as soon as possible).
-Scott
7:52 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
On Wed, Jul 21, 2004 at 10:52:46AM +0200, Ingo Molnar
wrote:
well, i share your desire for lower latencies but this is what the
semantics of softirqs require.
but it's really easy to break out of softirq processing early and
restart it later. All it needs in net_rx_action() is to:
if (softirq_defer && need_resched())
goto softnet_break;
since the network softirq code is already latency-aware. The same
'lock-break'-alike methods can be used for other softirq code too.
Ingo
this, if enabled, causes all softirqs to be
processed within ksoftirqd,
and it also breaks out of the softirq loop if preemption of ksoftirqd
has been triggered by a higher-prio task.
You'll still have the latency of finishing the currently executing
softirq, which often includes a loop itself (whose break condition is
based on not hogging the CPU, rather than letting higher priority
tasks in as soon as possible).
4:45 p.m.
On Wed, Jul 21, 2004 at 10:22:18AM +0200, Ingo Molnar wrote:
we already 'daemonize' softirqs in the stock
kernel if the load is high
enough. (this is what ksoftirqd does) So the only question is a tunable
to make this deferring of softirq load mandatory. Yarroll's patch is
quite complex, i dont think that is necessary.
What aspects of it do you find unnecessary? The second thread is
needed to maintain the current high/low priority semantics (without
that, you'll either starve regular tasks with a lot of softirqs, or
starve softirqs with a busy userspace, depending on how you set the
priority of the softirq thread).
It also has at least one
conceptual problem, the NOP-ing of local_bh_disable/enable in case of
CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?
Why is it "clearly wrong"? As far as I can tell, the only legitimate
use of it currently is to protect against deadlock (as in
spin_lock_bh()), which is not an issue if all softirqs run from a
thread. Ksoftirqd already ignores such disabling (unless I'm missing
something?), so code that uses it to synchronize with a softirq is
already broken.
There's also the possibility of code relying on it also being
preempt_disable(); if that's happening, it could be left alone
(though IMHO it'd be better if such code made its dependence on such
behavior explicit), with preempt_disable() being the only real
effect.
I've added a very simple solution to daemonize
softirqs similar to
Yarroll's patch to the -H5 version of voluntary-preempt:
BTW, it was my patch; Yarroll only submitted it to the list (as he
stated at the time).
-Scott
7:16 p.m.
On Wed, Jul 21, 2004 at 02:30:10PM -0400, Scott Wood wrote:
Why is it "clearly wrong"? As far as I can
tell, the only legitimate
use of it currently is to protect against deadlock (as in
spin_lock_bh()), which is not an issue if all softirqs run from a
thread. Ksoftirqd already ignores such disabling (unless I'm missing
something?), so code that uses it to synchronize with a softirq is
already broken.
It appears, though, that recent kernel versions do preempt_disable()
in ksoftirqd, apparently to support CPU hotplugging[1]. When I
originally made the patch (against 2.6.0), this wasn't the case.
Since it was done so recently, hopefully there are no cases since
then that have started depending on this behavior.
If preempt-disabled softirqs (and thus a local_bh_disable() that
works for mutual exclusion on the local CPU) become relied upon by
random pieces of kernel code, the ability to achieve very low
latencies in Linux (not counting Linux bolt-ons like RT-Linux) would
be substantially impared. At best, we'd have to use a per-CPU mutex
to replace local_bh_disable(), causing possible performance
degradation; this would require that any other locks held at the time
are also turned into mutexes.
-Scott
[1] Supporting this along with non-preempt-disable softirqs would
require some minor changes in the way that per-CPU softirqs are done
(especially in the network code). We did these changes in our 2.4
kernel, so as to allow priority inheritance to move a bound task
across CPUs[2]. In particular, the softirq needs to remember which CPU
it's processing data for, rather than using smp_processor_id() all
over the place, and there are a few places where a per-CPU spinlock
(only contended when the CPU binding is broken) needs to be acquired
(all of very short duration, at least in the 2.4 code).
Alternatively, some sort of "CPU reference count" could be used to
withhold permission to take a CPU offline without stopping
preemption.
[2] ...which is required for priority inheritance to work properly in
the presence of bound tasks without violating schedulability analysis
done on other CPUs.
7:35 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
It appears, though, that recent kernel versions do
preempt_disable()
in ksoftirqd, apparently to support CPU hotplugging[1]. When I
originally made the patch (against 2.6.0), this wasn't the case.
Since it was done so recently, hopefully there are no cases since then
that have started depending on this behavior.
do_softirq() always did a local_bh_disable() which stops preemption, so
softirq processing was always non-preemptible.
If preempt-disabled softirqs (and thus a
local_bh_disable() that works
for mutual exclusion on the local CPU) become relied upon by random
pieces of kernel code, [...]
believe me, as someone who took part in the discussions that designed
softirqs years ago and cleaned up some of it later on, i can tell you
that this property of softirqs was and is fully intentional. It's not
just some side-effect that got relied on by random code - it was used
from day one on. E.g. it enables exclusion against softirq contexts
without having to use cli/sti.
trying to make softirqs preemptible surely wont fly for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.
Ingo
8:29 p.m.
On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar wrote:
trying to make softirqs preemptible surely wont fly
for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.
These folks are tring to make the entire kernel fully preemptable,
possibly, to handle arbitrary preemption at any point during the
execution. It's a noble task to make the kernel preemptable in that
way, but what I've seen is that the use of non-preemptive critical
sections commits all locks below it in the call/lock graph to also
be non-preemptive critical sections and therefore forcing the use
of traditional lock-break and other techniques to lower latency.
Adding preemption points helps with the problems, but isn't something
that can be guaranteed to have a certain latency within N numbers of
context switches and some rescheduling computations, etc...
IMO, this is something that the Linux community should think about
being friendly to or have some kind of consideration for the possibility
of this.
bill
8:47 p.m.
On Wed, Jul 21, 2004 at 03:14:23PM -0700, Bill Huey wrote:
On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar
wrote:
One thing that would preserve the correctness of this system might be to
use a per-CPU blocking lock to directly back the uses of local_bh_*
functions explicitly. This altered system would be functionally isomorphic,
but would permit preemption in bhs.
This is something I've been thinking about for a while so it could be
off track or wrong and I'm open (pleading) for correction.
It's a radical departure from the current "deferred-everthing" model
that Linux and many other general purpose OSes are currently using.
The overall performance penalty is unknown, but the possiblity of
lowering latency could still be very significant.
bill
trying to make softirqs preemptible surely wont
fly for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.
These folks are tring to make the entire kernel fully preemptable,
possibly, to handle arbitrary preemption at any point during the
execution. It's a noble task to make the kernel preemptable in that
way, but what I've seen is that the use of non-preemptive critical
sections commits all locks below it in the call/lock graph to also
be non-preemptive critical sections and therefore forcing the use
of traditional lock-break and other techniques to lower latency.
8:53 p.m.
On Wed, Jul 21, 2004 at 11:18:26PM +0200, Ingo Molnar wrote:
* Scott Wood <scott(a)timesys.com> wrote:
Hmm... I'm not sure how I missed that (probably by misreading the
local_irq_enable() that comes after it as a local_bh_enable()).
It appears, though, that recent kernel versions
do preempt_disable()
in ksoftirqd, apparently to support CPU hotplugging[1]. When I
originally made the patch (against 2.6.0), this wasn't the case.
Since it was done so recently, hopefully there are no cases since then
that have started depending on this behavior.
do_softirq() always did a local_bh_disable() which stops preemption, so
softirq processing was always non-preemptible. believe me, as someone who took part in the
discussions that designed
softirqs years ago and cleaned up some of it later on, i can tell you
that this property of softirqs was and is fully intentional. It's not
just some side-effect that got relied on by random code - it was used
from day one on. E.g. it enables exclusion against softirq contexts
without having to use cli/sti.
It'd be nice to use locks that only exclude the specific regions in
the softirqs that are needed, but that's a lot to change at this
point...
trying to make softirqs preemptible surely wont fly
for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.
If every such loop in every softirq is taken care of, that would work
(though only until someone adds a new softirq that forgets to check
for preemption). I don't see any such checks in either the transmit
or receive network softirqs in vanilla 2.6.7, though (are they in a
patch, or am I overlooking them?), much less in each individual
driver. There are checks for excessive work (where "excessive" is
not well defined in terms of actual time), but none for need_resched()
except in a few isolated places.
-Scott
22 Jul
22 Jul
8:20 a.m.
New subject: [linux-audio-dev] [patch] voluntary-preempt-2.6.8-rc2-H9
* Scott Wood <scott(a)timesys.com> wrote:
i've added an infrastructure for easy softirq lock-break and preemption
to the -H9 version of the voluntary-preempt patch:
http://redhat.com/~mingo/voluntary-preempt/voluntary-preempt-2.6.8-rc2-H9
in -H9 i've fixed the most important softirq latency sources:
- the RCU code
this one is new and is really bad - it can literally execute tens of
thousands of d_callback() functions within rcu_do_batch() causing
millisecs of delays - e.g. triggered by the 'du /' test on a box with
enough RAM. It affects UP just as much as SMP, in both preempt and
non-preempt mode as well.
- the timer code
no real latencies in practice but in theory if enough timers are set
to fire in the same jiffy it could be easily unbound.
- net TX/RX code
being the worst offender this had some throttling code already but it
didnt listen to resched requests. It does now.
it is really easy to do lock-break of softirqs, one only has to find a
place where it's safe to enable softirq processing and do a
cond_resched_softirq() call.
Ingo
trying to make
softirqs preemptible surely wont fly for 2.6 and it will
also overly complicate the softirq model. What's so terminally wrong
about adding preemption checks to the softirq paths? It should solve the
preemption problem for good. The unbound softirq paths are well-known
(mostly in the networking code) and already have preemption-alike
checks.
If every such loop in every softirq is taken care of, that would work
(though only until someone adds a new softirq that forgets to check
for preemption). I don't see any such checks in either the transmit
or receive network softirqs in vanilla 2.6.7, though (are they in a
patch, or am I overlooking them?), much less in each individual
driver. There are checks for excessive work (where "excessive" is not
well defined in terms of actual time), but none for need_resched()
except in a few isolated places.
21 Jul
21 Jul
7:49 p.m.
* Scott Wood <scott(a)timesys.com> wrote:
what high/low semantics do you mean, other than the ordering of softirq
sources? (which is currently implemented via the __do_softirq() loop
first looking at the highest prio softirq.) So splitting up ksoftirqd
into two pieces seems like a separate issue.
local_bh_disable() excludes all softirq processing. This means that such
a section must not be preempted. E.g. the networking layer manipulates
per-CPU lists from such sections, if you remove local_bh_disable() then
from the middle of such a section we could preempt into ksoftirqd which
would break the code.
ok - sorry about the misattribution!
Ingo
we already
'daemonize' softirqs in the stock kernel if the load is high
enough. (this is what ksoftirqd does) So the only question is a tunable
to make this deferring of softirq load mandatory. Yarroll's patch is
quite complex, i dont think that is necessary.
What aspects of it do you find unnecessary? The second thread is
needed to maintain the current high/low priority semantics (without
that, you'll either starve regular tasks with a lot of softirqs, or
starve softirqs with a busy userspace, depending on how you set the
priority of the softirq thread). It also has at
least one
conceptual problem, the NOP-ing of local_bh_disable/enable in case of
CONFIG_SOFTIRQ_THREADS is clearly wrong. Yarroll?
Why is it "clearly wrong"? As far as I can tell, the only legitimate
use of it currently is to protect against deadlock (as in
spin_lock_bh()), which is not an issue if all softirqs run from a
thread. There's also the possibility of code relying on it
also being
preempt_disable(); if that's happening, it could be left alone (though
IMHO it'd be better if such code made its dependence on such behavior
explicit), with preempt_disable() being the only real effect.
yes, that's how softirqs are used. The patch changes these semantics.
I've added
a very simple solution to daemonize softirqs similar to
Yarroll's patch to the -H5 version of voluntary-preempt:
BTW, it was my patch; Yarroll only submitted it to the list (as he
stated at the time).
22 Jul
22 Jul
12:47 a.m.
On Wed, Jul 21, 2004 at 08:43:08PM +0200, Ingo Molnar wrote:
It's OK; I just don't want him to be blamed for my bugs. :-)
-Scott
* Scott Wood <scott(a)timesys.com> wrote:
I meant the current split between immediate-context softirqs (which
are repesented in the patch by the high-priority ksoftirqd) and the
low-priority thread which is used to avoid starvation while allowing
softirqs to continue running if the system's otherwise more or less
idle.
What aspects of it do you find unnecessary? The
second thread is
needed to maintain the current high/low priority semantics (without
that, you'll either starve regular tasks with a lot of softirqs, or
starve softirqs with a busy userspace, depending on how you set the
priority of the softirq thread).
what high/low semantics do you mean, other than the ordering of softirq
sources? (which is currently implemented via the __do_softirq() loop
first looking at the highest prio softirq.) So splitting up ksoftirqd
into two pieces seems like a separate issue. BTW, it was my
patch; Yarroll only submitted it to the list (as he
stated at the time).
ok - sorry about the misattribution!
8:04 a.m.
* Scott Wood <scott(a)timesys.com> wrote:
ok, i understand what you are trying to do. I dont think it makes much
sense to preserve the throttling property of the current
immediate/ksoftirqd processing. It was really an ad-hoc way to keep
softirqs from monopolizing the CPU. The simplest solution for softirq
deferral is to push it all into ksoftirqd and then let users change the
priority/policy of ksoftirqd.
it might make sense to create separate threads for each softirq level
that exists currently:
HI_SOFTIRQ=0,
TIMER_SOFTIRQ,
NET_TX_SOFTIRQ,
NET_RX_SOFTIRQ,
SCSI_SOFTIRQ,
TASKLET_SOFTIRQ
but this doesnt solve the problem either, because the current softirq
splitup is too opaque - there's no priority-based distinction between
softirq processing. Doing full softirq scheduling by attaching the
softirq work to the process context that generates it (or an anymous
context for things that are not connected to any particular existing
context) is way too much work and not completely possible anyway. Much
of the irq work <-> context information is lost at higher levels. We
merge IO requests and do other optimizations. To track who generated
what would be quite some work.
so since we cannot do it perfectly i'd go for the simplest approach for
now: defer to a single ksoftirqd per CPU. Then if someone comes up with
a good patch to attach all hardirq/softirq processing to a particular
context we can implement precise scheduling of hardirq/softirq work,
based on the priority/policy of the context that generated/receives the
interrupt event.
Ingo
what high/low
semantics do you mean, other than the ordering of softirq
sources? (which is currently implemented via the __do_softirq() loop
first looking at the highest prio softirq.) So splitting up ksoftirqd
into two pieces seems like a separate issue.
I meant the current split between immediate-context softirqs (which
are repesented in the patch by the high-priority ksoftirqd) and the
low-priority thread which is used to avoid starvation while allowing
softirqs to continue running if the system's otherwise more or less
idle.
4:51 p.m.
On Thu, Jul 22, 2004 at 11:51:11AM +0200, Ingo Molnar wrote:
* Scott Wood <scott(a)timesys.com> wrote:
Ideally, yes. However, there isn't currently a scheduling policy that
allows the softirq thread to run as a moderately high priority
realtime thread for a short period of time, and drop it to a low
priority non-realtime thread if it runs for longer than it "should".
Running it as one high priority non-realtime task would work if all
you want to run are non-realtime tasks and very high priority RT
tasks (which are intended to have higher priority than softirqs).
This is assuming that a high-priority non-RT task will always preempt
a lower priority task except when it has depleted its timeslice; I'm
not familiar enough with the current scheduler to know whether that
is the case.
I meant the current split between
immediate-context softirqs (which
are repesented in the patch by the high-priority ksoftirqd) and the
low-priority thread which is used to avoid starvation while allowing
softirqs to continue running if the system's otherwise more or less
idle.
ok, i understand what you are trying to do. I dont think it makes much
sense to preserve the throttling property of the current
immediate/ksoftirqd processing. It was really an ad-hoc way to keep
softirqs from monopolizing the CPU. The simplest solution for softirq
deferral is to push it all into ksoftirqd and then let users change the
priority/policy of ksoftirqd. it might make sense to create separate threads for
each softirq level
that exists currently:
HI_SOFTIRQ=0,
TIMER_SOFTIRQ,
NET_TX_SOFTIRQ,
NET_RX_SOFTIRQ,
SCSI_SOFTIRQ,
TASKLET_SOFTIRQ
but this doesnt solve the problem either, because the current softirq
splitup is too opaque - there's no priority-based distinction between
softirq processing.
Splitting it that way would be nice from the perspective of giving
the user the ability to place more importance on certain types of
softirqs, but it's not what I was trying to do with the high/low
threads.
so since we cannot do it perfectly i'd go for the
simplest approach for
now: defer to a single ksoftirqd per CPU.
The dual-thread model is about as simple as the immediate/thread
split that is currently there, IMHO, and while not perfect, it is an
improvement over having to pick only one policy/priority given the
current choices of policy.
-Scott
5:25 p.m.
On Thu, 2004-07-22 at 14:36 -0400, Scott Wood wrote:
This is assuming that a high-priority non-RT task will
always preempt
a lower priority task except when it has depleted its timeslice; I'm
not familiar enough with the current scheduler to know whether that
is the case.
Yes, this is the basic scheduling rule for non-RT: the highest priority
task, with timeslice remaining, runs.
Robert Love
12 Jul
12 Jul
11:04 p.m.
Is this problem strictly with reiserfs in 2.6 or is it also a problem
with 2.4. I actually experienced fewer xruns with reiserfs vs ext3 on
2.4 (preempt/ll) but I have no hard numbers to back that up.
Jan
On Mon, 2004-07-12 at 19:17, Lee Revell wrote:
On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
Lee Revell <rlrevell(a)joe-job.com> wrote:
Gnutella client. Forgot about that. I agree, it is not reasonable to
expect low latency with this kind of network traffic happening. I am
impressed it worked as well as it did.
On Sun, 2004-07-11 at 01:25, Andrew Morton wrote:
OK, thanks. The problem areas there are the timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
What we need to do is to encourage audio testers
to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
OK, here goes. The following traces result from running JACK overnight
like so, on an otherwise idle system. Hardware is a VIA EPIA 6000, with
a 600Mhz C3 processor. Kernel is 2.6.7 + volunatary_preempt patch.
voluntary_preempt and kernel_preemption are both on.
jackd -v --realtime -d alsa --outchannels 2 --rate 48000 --shorts
--playback --period 32 --nperiods 2
These settings require less than 666 microseconds scheduler latency.
The average performance is quite good - 5-20 *microseconds*! resierfs: yes, it's a problem. I
"fixed" it multiple times in 2.4, but the
fixes ended up breaking the fs in subtle ways and I eventually gave up.
Interesting. There is an overwhelming consensus amongst Linux audio
folks that you should use reiserfs for low latency work. Should I try
ext3?
Lee
11:11 p.m.
(please don't top-post)
Jan Depner <eviltwin69(a)cableone.net> wrote:
Is this problem strictly with reiserfs in 2.6 or is it also a problem
with 2.4. I actually experienced fewer xruns with reiserfs vs ext3 on
2.4 (preempt/ll) but I have no hard numbers to back that up.
Actually, the 2.4 low-latency patch does still have some reiserfs fixes, so
it's probably better than reiserfs in 2.6.
13 Jul
13 Jul
1:41 a.m.
On Mon, 2004-07-12 at 17:17, Lee Revell wrote:
There is an overwhelming consensus amongst Linux
audio
folks that you should use reiserfs for low latency work.
I doubt the "overwhelming consensus" part. ReiserFS is good at
creating/deleting lots of small files and that's not a typical scenario
when doing audio work.
For audio stuff the FS does not seem to matter much. However, when
working with large files (there are audio apps that create large project
files sometimes, especially when working in 96/24 mode and that kind of
stuff) i've seen better results with XFS.
When files get really big (think: video capture) there's no doubt XFS is
best. I've had a video capture app skipping frames no matter what, on
Ext3 and ReiserFS. Switch to XFS, the result: no more frame skipping.
--
Florin Andrei
http://florin.myip.org/
4:41 a.m.
On Mon, Jul 12, 2004 at 08:31:18PM -0700, Florin Andrei wrote:
On Mon, 2004-07-12 at 17:17, Lee Revell wrote:
me too, but at some point in the 2.4 kernel cycle,
reiserfs came out much better than ext3 in some latency tests* that
were reported on linux-audio-dev and linux-audio-user lists.
This seems to have left many of us musicianly types with a vague
"reiser good, ext3 bad" mindset.
* i think it was this:
http://myweb.cableone.net/eviltwin69/Arcana.html#Mark%20Knecht's%20file…
--
Paul Winkler
http://www.slinkp.com
There is an overwhelming consensus amongst Linux
audio
folks that you should use reiserfs for low latency work.
I doubt the "overwhelming consensus" part.
11:34 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
Lee Revell wrote:
On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
Why not reasonable ? It is very important that networking and HD I/O
both don't interfere with low latency audio.
Think about large audio setups where you use PC hardware to act as
dedicated samplers, software synthesizers etc.
Such clusters might be diskless and communicate with a GBit ethernet
with a high performance file server and
in some cases lots of network I/O might occur during audio playback. So
having latency spikes during network I/O
would be a big showstopper for many apps in certain setups.
Even ardour if run on a diskless client would need low latency while
doing network I/O because it does lots of disk I/O
which on the diskless client translate to lots of network I/O.
Typical use could be educational or research institutions where diskless
clients drastically lower the cost of managing large number
of boxes and allow sharing of resources. See the LTSP project.
Other low latency while high network I/O uses are video conferencing
applications, diskless settop boxes that stream
videos over IP etc.
Therefore low level network kernel develeopers think about us poor real
time multimedia souls :)
cheers,
Benno
http://www.linuxsampler.org
OK, thanks. The problem areas there are the timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
Gnutella client. Forgot about that. I agree, it is not reasonable to
expect low latency with this kind of network traffic happening. I am
impressed it worked as well as it did.
2:41 p.m.
On Tue, Jul 13, 2004 at 10:50:38AM -0400, Paul Davis wrote:
Fileservering for audio playback/recording is (the only really big FS I
have in my studio is on an NFS server), and audio rendering clusters are a
reasonable requirement, though maybe not over generic ethernet based IP.
- Steve
Other low
latency while high network I/O uses are video conferencing
applications, diskless settop boxes that stream
videos over IP etc.
none of these are low latency applications.
3:53 p.m.
On Tue, Jul 13, 2004 at 10:50:38AM -0400, Paul Davis
wrote:
Fileservering for audio playback/recording is (the only really big FS I
have in my studio is on an NFS server), and audio rendering clusters are a
reasonable requirement, though maybe not over generic ethernet based IP.
totally agreed. when i said "none of these", i meant the ones listed
above. you can do significant buffering in those cases. you don't need
sub-msec scheduler latencies to handle them.
--p
Other low
latency while high network I/O uses are video conferencing
applications, diskless settop boxes that stream
videos over IP etc.
none of these are low latency applications.
5:55 p.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
On Tue, 2004-07-13 at 09:26, Benno Senoner wrote:
Lee Revell wrote:
Right, I did not think of these. I just meant that you would not
typically run Gnutella on a DAW. This should never interfere with audio
latency, as Gnutella is running at normal priority and JACK is running
SCHED_FIFO.
Besides, it looks like there is a real problem with the route cache
flushing. I ran the same test with no significant network traffic and I
am still seeing spikes in latency. There are all of 5 entries in the
route cache.
Kernel IP routing cache
Source Destination Gateway Flags Metric Ref Use Iface
vmail-mx.dca.ne 192.168.2.102 192.168.2.102 l 0 1 838 lo
192.168.2.102 vmail-mx.dca.ne . 0 0 4 eth0
vmail-mx.dca.ne 192.168.2.102 192.168.2.102 l 0 1 838 lo
192.168.2.102 . . 0 0 2 eth0
192.168.2.102 vmail-mx.dca.ne . 0 0 4 eth0
Jul 13 15:15:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 15:15:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 15:15:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 15:15:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 15:15:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 15:15:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:15:49 mindpipe kernel: [rt_run_flush+77/144] rt_run_flush+0x4d/0x90
Jul 13 15:15:49 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 13 15:15:49 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 13 15:15:49 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 13 15:15:49 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 13 15:15:49 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 15:15:49 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 15:15:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:15:49 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 13 15:15:49 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 13 15:15:49 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 13 15:15:49 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 13 15:15:49 mindpipe kernel:
Jul 13 15:25:49 mindpipe kernel: ALSA
/usr/src/alsa-cvs-1.0.5/alsa-driver/alsa-kernel/core/pcm_lib.c:169: XRUN: pcmC0D0p
Jul 13 15:25:49 mindpipe kernel: [__crc_totalram_pages+1387264/5353478]
snd_pcm_period_elapsed+0x2ca/0x410 [snd_pcm]
Jul 13 15:25:49 mindpipe kernel: [__crc_totalram_pages+1455070/5353478]
snd_emu10k1_interrupt+0x3c8/0x480 [snd_emu10k1]
Jul 13 15:25:49 mindpipe kernel: [handle_IRQ_event+49/96] handle_IRQ_event+0x31/0x60
Jul 13 15:25:49 mindpipe kernel: [do_IRQ+155/352] do_IRQ+0x9b/0x160
Jul 13 15:25:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:25:49 mindpipe kernel: [local_bh_enable+17/112] local_bh_enable+0x11/0x70
Jul 13 15:25:49 mindpipe kernel: [rt_run_flush+92/144] rt_run_flush+0x5c/0x90
Jul 13 15:25:49 mindpipe kernel: [rt_secret_rebuild+0/48] rt_secret_rebuild+0x0/0x30
Jul 13 15:25:49 mindpipe kernel: [rt_secret_rebuild+14/48] rt_secret_rebuild+0xe/0x30
Jul 13 15:25:49 mindpipe kernel: [run_timer_softirq+201/416]
run_timer_softirq+0xc9/0x1a0
Jul 13 15:25:49 mindpipe kernel: [__do_softirq+131/144] __do_softirq+0x83/0x90
Jul 13 15:25:49 mindpipe kernel: [do_softirq+53/64] do_softirq+0x35/0x40
Jul 13 15:25:49 mindpipe kernel: [do_IRQ+261/352] do_IRQ+0x105/0x160
Jul 13 15:25:49 mindpipe kernel: [common_interrupt+24/32] common_interrupt+0x18/0x20
Jul 13 15:25:49 mindpipe kernel: [default_idle+35/64] default_idle+0x23/0x40
Jul 13 15:25:49 mindpipe kernel: [apm_cpu_idle+165/384] apm_cpu_idle+0xa5/0x180
Jul 13 15:25:49 mindpipe kernel: [cpu_idle+45/64] cpu_idle+0x2d/0x40
Jul 13 15:25:49 mindpipe kernel: [start_kernel+392/464] start_kernel+0x188/0x1d0
Jul 13 15:25:49 mindpipe kernel:
On Mon, 2004-07-12 at 19:31, Andrew Morton wrote:
Why not reasonable ? It is very important that networking and HD I/O
both don't interfere with low latency audio.
Think about large audio setups where you use PC hardware to act as
dedicated samplers, software synthesizers etc.
OK, thanks. The problem areas there are the timer-based route cache
flushing and reiserfs.
We can probably fix the route caceh thing by rescheduling the timer after
having handled 1000 routes or whatever, although I do wonder if this is a
thing we really need to bother about - what else was that machine up to?
Gnutella client. Forgot about that. I agree, it is not reasonable to
expect low latency with this kind of network traffic happening. I am
impressed it worked as well as it did.
16 Jul
16 Jul
3 a.m.
On Sat, 2004-07-10 at 22:25, Andrew Morton wrote:
What we need to do is to encourage audio testers to
use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
Just out of curiosity:
If i enable CONFIG_SND_DEBUG when compiling the kernel, but do not set
/proc/*/xrun_debug, will that kernel be slower than without
CONFIG_SND_DEBUG turned on?
--
Florin Andrei
http://florin.myip.org/
10:59 a.m.
New subject: [linux-audio-dev] Re: [announce] [patch] Voluntary Kernel Preemption Patch
At Thu, 15 Jul 2004 21:50:29 -0700,
Florin Andrei wrote:
On Sat, 2004-07-10 at 22:25, Andrew Morton wrote:
XRUN is checked regardless with or without CONFIG_SND_DEBUG, so there
is no difference in the performance. It will just print out messages
at each XRUN.
With debug option, some debug checks will be compiled in, but the
impact to perfomance must be ignorable.
Takashi
What we need to do is to encourage audio testers
to use ALSA drivers, to
enable CONFIG_SND_DEBUG in the kernel build and to set
/proc/asound/*/*/xrun_debug and to send us the traces which result from
underruns.
Just out of curiosity:
If i enable CONFIG_SND_DEBUG when compiling the kernel, but do not set
/proc/*/xrun_debug, will that kernel be slower than without
CONFIG_SND_DEBUG turned on?
7420
days inactive
7473
days old
linux-audio-dev@lists.linuxaudio.org
178 comments
34 participants
participants (34)
-
Andrea Arcangeli -
Andrew McGregor -
Andrew Morton -
Arjan van de Ven -
Benno Senoner -
Bill Huey -
Con Kolivas -
Dave Robillard -
Eric St-Laurent -
Fernando Pablo Lopez-Lezcano -
Florian Schmidt -
Florin Andrei -
Ingo Molnar -
Jan Depner -
Jens Axboe -
John Stoffel -
Lee Revell -
Maarten de Boer -
Mark Lord -
Mikulas Patocka -
Nick Piggin -
Paul Davis -
Paul Winkler -
Pavel Machek -
Ralf Beck -
Robert Love -
Ryan Underwood -
Scott Wood -
Steve Harris -
Takashi Iwai -
Thomas Charbonnel -
Tim Orford -
Timothy Miller -
Zwane Mwaikambo