14 May
2003
14 May
'03
11:05 a.m.
is there anyone here who has any idea how to handle segv's in a
linuxthreads-using application? i know that many of you are aware that
i've used threads a lot in my apps, and grappled with lots of tricky
signal related issues. however, i continue to see something really
deeply ugly in linuxthreads: segfaults seem not to be handled in the
same way as other signals. when a thread in ardour causes a segfault,
that thread is killed (actually, it becomes a zombie, waiting for its
parent to exit) and thats the end of the signal handling. for all
other signals, my code causes them are delivered to a single thread
that is waiting on all waitable signals except SIGCHLD, and we get
control there in a useful way.
its very distressing: a segfault will cause ardour to cease normal
operation, but nothing knows it has happened. if its in (say) the disk
butler thread, the GUI will continue working normally, but nothing
will actually do the right thing.
has anybody come up with code in which segfaults within threads are
handled? i have a few ideas on an approach that might work, but i'm
asking first :)
nb. this is true in JACK as well.
14 May
14 May
12:23 p.m.
On Wednesday 14 May 2003 14.59, Paul Davis wrote:
[...]
has anybody come up with code in which segfaults
within threads are
handled? i have a few ideas on an approach that might work, but i'm
asking first :)
I have no idea why segfaults are treated specially, or how to trap
them, but if no documentation explains it, maybe the code in
ElectricFence or similar debugging tool does?
I believe the SDL parachute traps segfaults as well, BTW. I'll have a
look...
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
1:08 p.m.
On Wednesday 14 May 2003 16.12, David Olofson wrote:
[...]
I believe the SDL parachute traps segfaults as well, BTW. I'll have
a look...
It does indeed, but I can't find any special treatment. Looks like
just another signal.
However, according to 'man signal', it seems to me that SIGSEGV (and
some others) *are* a bit special. I concluded that you have to make
sure the thread that causes the segfault must be killed by the signal
handler, or weird things will happen. (Zombie, endless loop calling
the signal handler or something. My system seems to do the latter.)
This code Works Here(TM), or at least does what I think it should:
---8<------------------------------------------------------
#include <stdlib.h>
#include <stdio.h>
#include <signal.h>
#include <string.h>
#include <unistd.h>
#include <pthread.h>
void handler(int sig)
{
signal(sig, SIG_DFL);
switch (sig)
{
case SIGSEGV:
puts("We got a segfault!");
pthread_exit(NULL);
break;
default:
puts("Unknown");
break;
}
}
void *crasher(void *arg)
{
long i = 42, *j = NULL;
signal(SIGSEGV, handler);
puts("crasher() started.");
sleep(3);
puts("crasher(): Let's crash!");
*j = i;
puts("crasher() ended. (How the f...!?)");
}
int main(int argc, char *argv[])
{
pthread_t thread;
puts("main() started.");
pthread_create(&thread, NULL, crasher, NULL);
while(1)
{
puts("main() is alive.");
sleep(1);
}
return 0;
}
------------------------------------------------------>8---
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
1:48 p.m.
However, according to 'man signal', it seems to
me that SIGSEGV (and
some others) *are* a bit special. I concluded that you have to make
thats true. the deeper problem here is that every thread has blocked
SIGSEGV, except for the one that is waiting in sigwait. pthreads
semantics say that signal handlers are shared across threads, but that
masks are per-thread. hence, the thread that causes the segv should
*not* receive the signal - it should be delivered to the sigwait-ing
thread.
i looked at the source for linuxthreads, and i can only chalk this up
to another "error" in the implementation of linuxthreads.
the only reason the handling ever appears to work is that most signals
are things like SIGINT/SIGTERM from a keyboard-input ctrl-c. this is
sent by the tty/term code to the entire process group, and thus even
the sigwait-ing thread gets it.
but any signal generated by a given thread within linuxthreads will be
delivered to that thread, regardless of what P.1003 says. hopefully,
NGPT will be better in this respect.
for now, i've fixed things so that things don't get so "odd" when a
segv happens - the handler exits the entire program regardless of the
thread it is executing in.
--p
1:26 p.m.
On Wed, May 14, 2003 at 04:12:39PM +0200, David Olofson wrote:
> has anybody come up with code in which segfaults
within threads are
> handled? i have a few ideas on an approach that might work, but i'm
> asking first :)
...
I believe the SDL parachute traps segfaults as well,
BTW. I'll have a
look...
Yes, it does, I've noticed it doing that on meterbridge, which is a jack
app.
- Steve
2 p.m.
On Wednesday 14 May 2003 17.15, Steve Harris wrote:
On Wed, May 14, 2003 at 04:12:39PM +0200, David
Olofson wrote:
Yeah... However, I realized why it doesn't do anything special: It
cleans up and terminates the application before leaving the signal
handler, so it never runs into the "life after death" problem.
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
> has anybody come up with code in which
segfaults within threads
> are handled? i have a few ideas on an approach that might work,
> but i'm asking first :)
...
I believe the SDL parachute traps segfaults as
well, BTW. I'll
have a look...
Yes, it does, I've noticed it doing that on meterbridge, which is a
jack app. 
15 May
15 May
6:14 a.m.
On Wed, May 14, 2003 at 05:49:42PM +0200, David Olofson wrote:
[SDL Parachute on SEGV]
Yeah... However, I realized why it doesn't do
anything special: It
cleans up and terminates the application before leaving the signal
handler, so it never runs into the "life after death" problem.
If your users in a threaded environment have tried to corrupt
memory this is a very sensible thing you can do, since you cannot trust
your own address space anymore.
Re-exec is an alternative I could envision for realtime.
Regards
Ingo Oeser
6:46 a.m.
On Wednesday 14 May 2003 23.52, Ingo Oeser wrote:
On Wed, May 14, 2003 at 05:49:42PM +0200, David
Olofson wrote:
[SDL Parachute on SEGV]
[...]
Of course - and the best solution is to never have bugs in the first
place. ;-)
Seriously though, even if you're going to save data to an emergency
save file or something, this is the way to go. The more code you
execute before saving the data, the bigger the risk that whatever
happened before the segfault causing further data corruption.
Freeze - Save - Exit. :-)
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
Yeah... However, I realized why it doesn't do
anything special:
It cleans up and terminates the application before leaving the
signal handler, so it never runs into the "life after death"
problem.
If your users in a threaded environment have tried to corrupt
memory this is a very sensible thing you can do, since you cannot
trust your own address space anymore.
6:50 a.m.
On Wednesday 14 May 2003 23.52, Ingo Oeser wrote:
[...]
Re-exec is an alternative I could envision for
realtime.
Further thoughts on that:
If you have a setup with multiple redundant processes and/or machines,
the obvious job for the "crash handler" is to tell some other process
or machine to take over. (This could be a matter of sending some
commands to a hardware mixer or "router".) After that, you should
just get out of the way ASAP, and maybe restart.
//David Olofson - Programmer, Composer, Open Source Advocate
.- The Return of Audiality! --------------------------------.
| Free/Open Source Audio Engine for use in Games or Studio. |
| RT and off-line synth. Scripting. Sample accurate timing. |
`-----------------------------------> http://audiality.org -'
--- http://olofson.net --- http://www.reologica.se ---
5:11 p.m.
On Thu, May 15, 2003 at 10:39:45AM +0200, David Olofson wrote:
On Wednesday 14 May 2003 23.52, Ingo Oeser wrote:
[...]
If you are talking HA, then the other processes (or some kind of
monitor in a more fragile solution) should detect the failure and
take over.
If you crash, then you cannot trust your addressspace anymore.
Detecting a crash is very simple for the other process: Just have
a never to be used pipe open and detect SIGCHLD or SIGPIPE. Then
you even know, which one of processes died.
Cannot do that with threads? What a pity...
Anything more sophisticated, which involves data from the
segfaulting process asks for trouble.
Re-exec means, that you just trust your read-only segment, which
is a good assumption until hardware memory errors exist. If they
exist, you have other problems. I'm talking about exec called
with read-only constants, to clarify that.
Regards
Ingo Oeser
Re-exec is an alternative I could envision for
realtime.
Further thoughts on that:
If you have a setup with multiple redundant processes and/or machines,
the obvious job for the "crash handler" is to tell some other process
or machine to take over. (This could be a matter of sending some
commands to a hardware mixer or "router".) After that, you should
just get out of the way ASAP, and maybe restart. 
14 May
14 May
2:31 p.m.
Paul Davis <paul(a)linuxaudiosystems.com> writes:
is there anyone here who has any idea how to handle
segv's in a
linuxthreads-using application? i know that many of you are aware that
i've used threads a lot in my apps, and grappled with lots of tricky
signal related issues. however, i continue to see something really
deeply ugly in linuxthreads: segfaults seem not to be handled in the
same way as other signals. when a thread in ardour causes a segfault,
that thread is killed (actually, it becomes a zombie, waiting for its
parent to exit) and thats the end of the signal handling. for all
other signals, my code causes them are delivered to a single thread
that is waiting on all waitable signals except SIGCHLD, and we get
control there in a useful way.
According to the O'Reilly] Pthreads (Caterpillar) book, signals due to
system exceptions (like SIGFPE, or SIGSEGV) are "synchronous" and
*must* be delivered to the offending thread. Quoting...
"It will be that thread, in its own context, that will handle the
signal on behalf of the process as a whole."
Of course, the default action for these signals is termination. But,
what happens if the offending thread tries to mask or ignore a
synchronous signal?
I suspect something strange here. I've seen crashes where the JACK
thread has disappeared before the core dump was written. Could they
have been triggered by a segfault in the process() thread?
its very distressing: a segfault will cause ardour to
cease normal
operation, but nothing knows it has happened. if its in (say) the disk
butler thread, the GUI will continue working normally, but nothing
will actually do the right thing.
has anybody come up with code in which segfaults within threads are
handled? i have a few ideas on an approach that might work, but i'm
asking first :)
No, but I'm interested. Right now, I'd be happy just to reliably get
a dump with the backtrace stack of the offending thread intact.
Sometimes that happens, other times not.
--
Jack O'Quin
Austin, Texas, USA
2:59 p.m.
According to the O'Reilly] Pthreads (Caterpillar)
book, signals due to
system exceptions (like SIGFPE, or SIGSEGV) are "synchronous" and
*must* be delivered to the offending thread. Quoting...
"It will be that thread, in its own context, that will handle the
signal on behalf of the process as a whole."
thats a very useful thing to know. its not mentioned in the book i use
on threads (kleiman/shah/smaalders). they mention the sync/async
distinction, but not the requirement that they be handled by the
thread that caused them.
i wonder if the kernel is enforcing the sync rule? i don't see how it
could, though - it doesn't know about linuxthreads. we've blocked
SIGSEGV, yet the thread still receives it. strange. maybe you can't
block SIGSEGV? why wouldn't sigaction fail?
I suspect something strange here. I've seen
crashes where the JACK
thread has disappeared before the core dump was written. Could they
have been triggered by a segfault in the process() thread?
unless the kernel has been fixed since i last checked on this (and i
have a vague memory that it has been), linux doesn't write core files
for multi-threaded processes.
No, but I'm interested. Right now, I'd be
happy just to reliably get
a dump with the backtrace stack of the offending thread intact.
Sometimes that happens, other times not.
i'll try to fix JACK so that it does the same thing as ardour is doing
now (exiting from within the handler).
--p
3:49 p.m.
Paul Davis <paul(a)linuxaudiosystems.com> writes:
I suspect that the reason there's no good documentation on signal
interactions this is because the pthreads design sucks in that area.
The pthreads workgroup was never able to graft two such different
concepts as threads and UNIX signals together cleanly. Maybe it can't
be done.
"It will
be that thread, in its own context, that will handle the
signal on behalf of the process as a whole."
thats a very useful thing to know. its not mentioned in the book i use
on threads (kleiman/shah/smaalders). they mention the sync/async
distinction, but not the requirement that they be handled by the
thread that caused them. i wonder if the kernel is enforcing the sync rule? i
don't see how it
could, though - it doesn't know about linuxthreads.
I think it has to for POSIX compliance.
IIUC, the kernel sees each thread as a lightweight process. It can
insist on either delivering the signal or terminating that
(lightweight) process. I think that's what AIX did.
unless the kernel has been fixed since i last checked
on this (and i
have a vague memory that it has been), linux doesn't write core files
for multi-threaded processes.
That was true once, but no longer. It dumps for me (with 2.4.19), but
maybe not always reliably for threads.
What may be confusing the issue is that lots of distributions
(including Debian) now eliminate all core dumps by set "ulimit -c" to
zero out of the box. I'm explicitly setting "ulimit -c 100000".
i'll try to fix JACK so that it does the same
thing as ardour is doing
now (exiting from within the handler).
Good idea.
If you send me a patch or put it in CVS, I'll give it a try.
--
Jack O'Quin
Austin, Texas, USA
7660
days inactive
7661
days old
linux-audio-dev@lists.linuxaudio.org
12 comments
5 participants
participants (5)
-
David Olofson -
Ingo Oeser -
Jack O'Quin -
Paul Davis -
Steve Harris