26 Jun
2006
26 Jun
'06
10:51 a.m.
Ben Collins wrote:
On Mon, 2006-06-26 at 10:11 +0200, Pieter Palmers
wrote:
I'm sorry for my previous response... a
monday-morning-bad-temper-leave-me-alone one that was very
non-constructive. On top of that it is quite stupid to ask for help and
at the same time claim "you'll have to take my word on it".
Please accept my apologies.
Lee Revell wrote:
Take damn a digital photo. I'm sorry for being a jerk, but I'm not going
to debug an oops blind :P On Mon, 2006-06-26 at 01:08 +0200, Pieter Palmers
wrote:
no :p. I'm sorry for being a jerk, but I'm not going to type over that
message just so that you can confirm that it indeed is a 'soft lockup'
(or whatever it is called exactly) and that it occurs in the
ohci1394_iso_unregister_tasklet. You'll have to take my word on it. If
you need some specific part of the kernel message, you can get it. Tell
me what you wqant and why, that way I can learn something from it. Hi,
We are experiencing 'soft' deadlocks when running our code (Freebob,
i.e. userspace lib for firewire audio) on RT kernels. After a few
seconds, I get a kernel panic message that signals a soft lockup.
Can we see the kernel panic message? ;-)
Seriously, if you are going to ask for help, be prepared to provide the
info requested, or plan on getting little to no help.
Of course. My monday-morning bad temper is over by now, and I hope I
didn't transfer it to any of you. I'll provide the panic, one way or
another.
Again, I apologize,
Pieter
26 Jun
26 Jun
11:30 a.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers wrote:
Ben Collins wrote:
> On Mon, 2006-06-26 at 10:11 +0200, Pieter Palmers wrote:
>> Lee Revell wrote:
Can we please use "Reply to All" for all followups, as is the convention
in kernel development? It's annoying to follow a thread when some of it
is in my Inbox, some is on LAD, and some apparently was mailed
privately ;-)
Lee
11:57 a.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers wrote:
Of course. My monday-morning bad temper is over by now, and I hope I
didn't transfer it to any of you. I'll provide the panic, one way or
another.
Can you reproduce the problem on a non-RT kernel?
Lee
3:05 p.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
Lee Revell wrote:
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers
wrote:
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
Greets & thanks,
Pieter
Of course. My monday-morning bad temper is over by now, and I hope I
didn't transfer it to any of you. I'll provide the panic, one way or
another.
Can you reproduce the problem on a non-RT kernel?
3:31 p.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 21:05 +0200, Pieter Palmers wrote:
Lee Revell wrote:
Can you add debug printk's before and after tasklet_kill() in
ohci1394_unregister_iso_tasklet to see where it locks up?
Lee
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers
wrote:
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
Of course. My monday-morning bad temper is over by now, and I hope I
didn't transfer it to any of you. I'll provide the panic, one way or
another.
Can you reproduce the problem on a non-RT kernel?
3:44 p.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
Lee Revell wrote:
On Mon, 2006-06-26 at 21:05 +0200, Pieter Palmers
wrote:
That's the first thing I did: the printk before tasklet_kill succeeds,
the one right after the tasklet_kill doesn't.
Pieter
Lee Revell wrote:
Can you add debug printk's before and after tasklet_kill() in
ohci1394_unregister_iso_tasklet to see where it locks up?
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers
wrote:
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
Of course. My monday-morning bad temper is over by now, and I hope I
didn't transfer it to any of you. I'll provide the panic, one way or
another.
Can you reproduce the problem on a non-RT kernel?
3:58 p.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 21:44 +0200, Pieter Palmers wrote:
Lee Revell wrote:
OK that's what I suspected.
It seems that the -rt patch changes tasklet_kill:
Unpatched 2.6.17:
void tasklet_kill(struct tasklet_struct *t)
{
if (in_interrupt())
printk("Attempt to kill tasklet from interrupt\n");
while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
do
yield();
while (test_bit(TASKLET_STATE_SCHED, &t->state));
}
tasklet_unlock_wait(t);
clear_bit(TASKLET_STATE_SCHED, &t->state);
}
2.6.17-rt:
void tasklet_kill(struct tasklet_struct *t)
{
if (in_interrupt())
printk("Attempt to kill tasklet from interrupt\n");
while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
do
msleep(1);
while (test_bit(TASKLET_STATE_SCHED, &t->state));
}
tasklet_unlock_wait(t);
clear_bit(TASKLET_STATE_SCHED, &t->state);
}
You should ask Ingo & the other -rt developers what the intent of this
change was. Obviously it loops forever waiting for the state bit to
change.
Lee
On Mon, 2006-06-26 at 21:05 +0200, Pieter Palmers
wrote:
That's the first thing I did: the printk before tasklet_kill succeeds,
the one right after the tasklet_kill doesn't. Lee Revell wrote:
Can you add debug printk's before and after tasklet_kill() in
ohci1394_unregister_iso_tasklet to see where it locks up?
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers
wrote:
>
> Of course. My monday-morning bad temper is over by now, and I hope I
> didn't transfer it to any of you. I'll provide the panic, one way or
> another.
>
Can you reproduce the problem on a non-RT kernel?
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
4:15 p.m.
New subject: [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 21:44 +0200, Pieter Palmers wrote:
Lee Revell wrote:
Actually the problem might not be the change to tasklet_kill() but the
change to tasklet_unlock_wait().
include/linux/interrupt.h:
#if defined(CONFIG_SMP) || defined(CONFIG_PREEMPT_RT)
static inline void tasklet_unlock_wait(struct tasklet_struct *t)
{
while (test_bit(TASKLET_STATE_RUN, &(t)->state)) { barrier(); }
}
#else
# define tasklet_unlock_wait(t) do { } while (0)
Can you add a printk before that while loop?
Lee
On Mon, 2006-06-26 at 21:05 +0200, Pieter Palmers
wrote:
That's the first thing I did: the printk before tasklet_kill succeeds,
the one right after the tasklet_kill doesn't. Lee Revell wrote:
Can you add debug printk's before and after tasklet_kill() in
ohci1394_unregister_iso_tasklet to see where it locks up?
On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers
wrote:
>
> Of course. My monday-morning bad temper is over by now, and I hope I
> didn't transfer it to any of you. I'll provide the panic, one way or
> another.
>
Can you reproduce the problem on a non-RT kernel?
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
4:35 p.m.
New subject: [Freebob-devel] [linux-audio-dev] ieee1394 deadlock on RT kernels
Lee Revell wrote:
On Mon, 2006-06-26 at 21:44 +0200, Pieter Palmers
wrote:
because you are not allowed to yield() in an RT context?
I wish I had been a little more elaborate on my initial mail, as it
would have saved us some time, and communication troubles (on my part
that is). I already spotted the msleep() change in the patch, and I
already tried reverting it. That gives you a nice new panic message,
something like 'BUG: yield()'ing in ...'.
I'm wondering why a patched, but not 'complete preemption' configured
kernel works fine. This change is present in them too, so it probably
has something to do with the msleep() implementation.
Another strange thing is: why doesn't the tasklet finish, so that it can
be 'unscheduled'? I have my IRQ priorities higher than any other RT
threads, so I would expect that the tasklet can finish. Or is
tasklet_kill not-preemtible? that would be very strange as I would
expect that busy waiting on something in a non-preemptible code path on
a single-cpu system always deadlocks.
Greets,
Pieter
Lee Revell wrote:
OK that's what I suspected.
It seems that the -rt patch changes tasklet_kill:
Unpatched 2.6.17:
void tasklet_kill(struct tasklet_struct *t)
{
if (in_interrupt())
printk("Attempt to kill tasklet from interrupt\n");
while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
do
yield();
while (test_bit(TASKLET_STATE_SCHED, &t->state));
}
tasklet_unlock_wait(t);
clear_bit(TASKLET_STATE_SCHED, &t->state);
}
2.6.17-rt:
void tasklet_kill(struct tasklet_struct *t)
{
if (in_interrupt())
printk("Attempt to kill tasklet from interrupt\n");
while (test_and_set_bit(TASKLET_STATE_SCHED, &t->state)) {
do
msleep(1);
while (test_bit(TASKLET_STATE_SCHED, &t->state));
}
tasklet_unlock_wait(t);
clear_bit(TASKLET_STATE_SCHED, &t->state);
}
You should ask Ingo & the other -rt developers what the intent of this
change was. Obviously it loops forever waiting for the state bit to
change.
On Mon, 2006-06-26 at 21:05 +0200, Pieter Palmers
wrote:
That's the first thing I did: the printk before tasklet_kill succeeds,
the one right after the tasklet_kill doesn't. Lee Revell wrote:
> On Mon, 2006-06-26 at 16:51 +0200, Pieter Palmers wrote:
>>
>> Of course. My monday-morning bad temper is over by now, and I hope I
>> didn't transfer it to any of you. I'll provide the panic, one way or
>> another.
>>
> Can you reproduce the problem on a non-RT kernel?
>
No, it only occurs with RT kernels, and only with those configured for
PREEMPT_RT. If I use PREEMPT_DESKTOP, there is no problem. (with
threaded IRQ's etc... only switched over the preemption level in the
kernel config).
I've uploaded the photo's of the panic here:
http://freebob.sourceforge.net/old/img_3378.jpg (without flash)
http://freebob.sourceforge.net/old/img_3377.jpg (with flash)
both are of suboptimal quality unfortunately, but all info is readable
on one or the other.
Can you add debug printk's before and after
tasklet_kill() in
ohci1394_unregister_iso_tasklet to see where it locks up?
5:05 p.m.
New subject: [Freebob-devel] [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 22:35 +0200, Pieter Palmers wrote:
because you are not allowed to yield() in an RT
context?
I wish I had been a little more elaborate on my initial mail, as it
would have saved us some time, and communication troubles (on my part
that is). I already spotted the msleep() change in the patch, and I
already tried reverting it. That gives you a nice new panic message,
something like 'BUG: yield()'ing in ...'.
I'm wondering why a patched, but not 'complete preemption' configured
kernel works fine. This change is present in them too, so it probably
has something to do with the msleep() implementation.
Because I suspect that the change to tasklet_unlock_wait() was actually
responsible.
Did you add a printk() yet as I suggested in my last mail?
Once you add a printk to verify that it hangs in the while loop in
tasklet_unlock_wait() I think you have enough information for a bug
report.
Lee
27 Jun
27 Jun
11:23 a.m.
New subject: [Freebob-devel] [linux-audio-dev] ieee1394 deadlock on RT kernels
On Mon, 2006-06-26 at 22:35 +0200, Pieter Palmers wrote:
Another strange thing is: why doesn't the tasklet
finish, so that it
can be 'unscheduled'? I have my IRQ priorities higher than any other
RT threads, so I would expect that the tasklet can finish. Or is
tasklet_kill not-preemtible? that would be very strange as I would
expect that busy waiting on something in a non-preemptible code path
on a single-cpu system always deadlocks.
When are you going to report this to Ingo + LKML + the other -rt
developers?
Lee
3:20 p.m.
New subject: [Freebob-devel] [linux-audio-dev] ieee1394 deadlock on RT kernels
Lee Revell wrote:
On Mon, 2006-06-26 at 22:35 +0200, Pieter Palmers
wrote:
After I do the printk testing to pinpoint the problem a little more
precise (as you suggested yesterday).
However, I didn't feel like iterating through the recompile
kernel/crash/reboot cycle even more yesterday.
Is there any underlying reason for this question?
Pieter
Another strange thing is: why doesn't the
tasklet finish, so that it
can be 'unscheduled'? I have my IRQ priorities higher than any other
RT threads, so I would expect that the tasklet can finish. Or is
tasklet_kill not-preemtible? that would be very strange as I would
expect that busy waiting on something in a non-preemptible code path
on a single-cpu system always deadlocks.
When are you going to report this to Ingo + LKML + the other -rt
developers?
3:30 p.m.
New subject: [Freebob-devel] [linux-audio-dev] ieee1394 deadlock on RT kernels
On Tue, 2006-06-27 at 21:20 +0200, Pieter Palmers wrote:
Lee Revell wrote:
Only that you seem to have found a bug in -rt and I'd like to get it
fixed.
Lee
On Mon, 2006-06-26 at 22:35 +0200, Pieter Palmers
wrote:
After I do the printk testing to pinpoint the problem a little more
precise (as you suggested yesterday).
However, I didn't feel like iterating through the recompile
kernel/crash/reboot cycle even more yesterday.
Is there any underlying reason for this question? Another strange thing is: why doesn't the
tasklet finish, so that it
can be 'unscheduled'? I have my IRQ priorities higher than any other
RT threads, so I would expect that the tasklet can finish. Or is
tasklet_kill not-preemtible? that would be very strange as I would
expect that busy waiting on something in a non-preemptible code path
on a single-cpu system always deadlocks.
When are you going to report this to Ingo + LKML + the other -rt
developers?
6544
days inactive
6545
days old
linux-audio-dev@lists.linuxaudio.org
12 comments
2 participants
participants (2)
-
Lee Revell -
Pieter Palmers