26 Jul
2009
26 Jul
'09
7:24 p.m.
On Sat, 25 Jul 2009, Brent Busby wrote:
In addition to the question I posted earlier about
PREEMPT_RCU, I've
found still other kernel config options that are not covered in most of
the extant howtos for setting up low latency kernels, because they are
recently added in the unpatched kernel. (Or at least, they're more
recent than most of the howtos...)
I've been researching these options, mostly by googling (and
googling...and googling) the Linux Kernel Mailing List archives, and
also by looking at the config used by 64Studio, since it seems the
prevalent opinion is that they are stable as a rock. With the latter
approach of checking against 64Studio's config, I've had little luck
though, because their distro currently comes in a 2.1 "stable" version
that is based on Debian "etch", with a 2.6.21 kernel that predates the
existence of some of the config options I'm asking about; and a 3.0
"beta" version that uses a newer kernel, but should also probably be
taken with a grain of salt, because 64Studio is still working the bugs
out of it, and because it seems to have options turned on that the LKML
archives have warned strongly against. So, most of the information I've
gleaned has been from the LKML.
I tried out the 64Studio stable version last fall, and had a lot of troubles
with it. The 2.6.21 based kernel wouldn't even boot on my box, so I ended up
booting up with the 2.6.24 based -rt kernel that I was using on Fedora 8 /
CCRMA. There were too many libraries and apps that were way too out of date
for my tastes, so I abandoned the whole experiment.
I might as well collate these questions here, for what
good it may do.
The kernel I'm configuring is 2.6.29.6 with matching RT patches (-rt23),
though all of the options listed here are actually present even in the
unpatched 2.6.29.6 kernel.
PREEMPT_RCU :
This was the one my original question was about in the earlier post.
In the newer kernels (even mainline ones, without any RT patches), there
is a choice of "RCU Subsystem", with one option being "classic", and
other being "preemptable". The choice would seem obvious for low
latency, except that the help text warns that a preemptible RCU is
likely to expose serious kernel bugs that may render the system
completely unstable. (This is *not* the same setting as the various
CONFIG_PREEMPT options that have been present in the mainline kernel for
awhile now. This is something new, and it appears in menuconfig as a
separate setting.) I think I've mostly gathered from reading through
the whole process of debugging that the kernel gurus went through that
as of last year or so, this is now considered mostly stable, after
having been subjected to a utility called 'rcutorture', and having fixed
many lockups. I'd still be interested in anything anyone else can
contribute about it though, especially if they're using it and they
also think it is stable.
There was a time when PREEMPT_RCU was unstable, and most of the -rt kernel
releases would fix one RCU bug while adding another. These times are now past,
however. PREEMPT_RCU has been completely stable for me on AMD and Intel (both
multicore) for about a year now.
GROUP_SCHED :
This one is interesting, and I don't know what to make of it, other than
that the LKML seems to have decided in the last two months or so that it
slows your system down and makes latencies worse.
The thing that's confusing about it though is that it's
described as a mechanism for grouping high priority tasks by group.
It's implied (though not spelled out specifically) that they even mean
by this Posix groups, because in one document I read, it says that
enabling this will cause you to be unable to get realtime as a non-root
user unless you are setup in a group specified in limits.conf. Hmm!
That sounds an awful lot like what we've just been calling pam_limits
for years now. Are we doing this with a kernel config option now? (One
which apparently doesn't work?)
The 3.0 (beta) version of 64Studio turns this on. Then again,
looking at their kernel config, they seem to turn everything on, and
that might be why it's considered beta. (The 2.1 stable version of
64Studio seems to have a kernel old enough that it never had that
option as a choice.) Anyone have any feedback about this? It's only in
the past two months that the kernel gurus have decided that it's bad,
but they're actually considering marking it broken from what I've read.
Here's another vote for completely broken! Every time I set up a box with the
-rt kernel, I start with the distro's .config and tweak from there. And every
time, I can't make realtime low-latency audio a reality until I disable
GROUP_SCHED.
CGROUPS :
This sounds cool, but I'm reasonably sure it's not actually necessary.
Documentation I've read suggests that this allows for letting
applications (or users) define CPU and memory pool affinity for tasks,
so that one could arbitrarily tie down particular threads or tasks to a
given processor core, or region of memory (or something like that).
However, the thing that makes me somewhat sure I don't positively need
this is that the same documentation also says that if you want to use
it, you need to create a new subdirectory under /dev, mount a new
pseudofilesystem under it, and then this module will populate that space
with dynamic configuration data about these affinity groups for running
tasks. I have neither seen any distro (even ones made for musicians)
that has set any such thing up out of the box, nor have I ever seen a
realtime howto that tells people to do it themselves. It sounds like
there's a lot of infrastructure necessary for this that's not common in
distros yet. (Though I see that regular Debian "lenny" turns this on in
the kernel without actually providing the special /dev support for it,
which I presume they're thinking you'll setup yourself if you're
interested.) Still, comments welcome...
As far as I can tell, CGROUPS doesn't affect realtime latencies enough to make
any sort of difference for realtime audio. It does, however, mess with the
scheduler, even without setting up the affinity groups. I've noted some
strange scheduling behavior: When running multiple instances of the same CPU
bound task (like burnP6, mprime, dd, or your favorite stress tester), the tasks
usually start off fairly evenly distributed between the cores, and then the
affinity seems to drift, lumping these tasks on one half of the cores while
leaving the other half mostly idle. On a core2quad with CGROUPS enabled, I
generally have to run 12 instances of burnP6 to keep utilization of all four
cores pegged at 99-100%. I wouldn't go as far as to say that CGROUPS is
broken, but I fail to see the usefulness on a production audio system with the
IRQ and other realtime priorities tuned properly, especially now that the core
scheduling features for -rt are stable.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
I've also run into a handful of config options that either help or hurt
realtime performance, depending on the specific hardware:
NO_HZ (seems to work properly on most hardware these days)
PCI_MSI (some PCI devices seem to add more latency with MSI)
RTC_DRV_CMOS (this depends on your particluar RTC hardware)
HPET_EMULATE_RTC (this depends on your particluar HPET hardware)
X86_ACPI_CPUFREQ (creates TSC drift between cores on Intel)
And of course, there's all kinds of profiling, stats, testing, and debugging
options that make any realtime load suffer. Occasionally, you'll find a device
driver that's just plain bad for -rt performance, but I've been running into
this problem a lot less these days.
Equally as important for low-latency are the scheduling priorities. After a lot
of google, lkml, and trial and error, I've settled on the following for
rock-solid low-latency audio:
99 FF migration
99 FF posixcputmr
98 FF IRQ-8 (realtime clock)
97 FF audio IRQ (in some cases means ieee1394 or specific USB port)
80 RR JACK
All audio and MIDI apps should run at a lower realtime priority than JACK. All
other IRQ- and sirq- threads should be set to
SCHED_OTHER. To set this up, I've added the following to my /etc/rc.local:
---
# set all irq threads to sched_other
for irq in `pgrep 'IRQ-'`; do
chrt -o -p 0 $irq;
done
# set all softirq threads to sched_other
for sirq in `pgrep 'sirq-'`; do
chrt -o -p 0 $sirq;
done
# set high prio IRQs
chrt -f -p 98 `pgrep IRQ-8` # rtc
chrt -f -p 97 `pgrep IRQ-16` # hda-intel
---
And of course, there's those pesky BIOS options. Some of the newer Intel CPU
features like to generate interrupts that the kernel can do absolutely nothing
about. I had to disable C1E, CPU TM function, execute disable bit, and one of
the ACPI options (can't remember which one) to get my core2quad to work.
Disabling video interrupts (only an issue on older hardware) always seems to
help. AMD systems seem to require much less (if any) BIOS tweaking for -rt.
Please, anyone, correct me if I'm wrong on any of this. Most of this knowledge
is gained from four years of following -rt development and tuning -rt kernels
on a variety of hardware. I'm always adding to my bag of tricks when it comes
to tuning realtime machines ;-}
Cheers,
--ww
26 Jul
26 Jul
10:03 p.m.
New subject: More recent kernel config options
On Sun, Jul 26, 2009 at 02:24:25PM -0700, William Weston wrote:
On Sat, 25 Jul 2009, Brent Busby wrote:
Is that advised? I was told to use HZ_1000 in order to get MIDI to work correctly. Should
I be using NO_HZ instead? I've got several machines to configure: a 32-bit Atom EEE, a
64-bit Intel Micro-ITX, and a 64-bit Asus Intel Core2Duo
In addition to the question I posted earlier
about PREEMPT_RCU, I've
found still other kernel config options that are not covered in most of
the extant howtos for setting up low latency kernels, because they are
recently added in the unpatched kernel. (Or at least, they're more
recent than most of the howtos...)
I've been researching these options, mostly by googling (and
googling...and googling) the Linux Kernel Mailing List archives, and
also by looking at the config used by 64Studio, since it seems the
prevalent opinion is that they are stable as a rock. With the latter
approach of checking against 64Studio's config, I've had little luck
though, because their distro currently comes in a 2.1 "stable" version
that is based on Debian "etch", with a 2.6.21 kernel that predates the
existence of some of the config options I'm asking about; and a 3.0
"beta" version that uses a newer kernel, but should also probably be
taken with a grain of salt, because 64Studio is still working the bugs
out of it, and because it seems to have options turned on that the LKML
archives have warned strongly against. So, most of the information I've
gleaned has been from the LKML.
I tried out the 64Studio stable version last fall, and had a lot of troubles
with it. The 2.6.21 based kernel wouldn't even boot on my box, so I ended up
booting up with the 2.6.24 based -rt kernel that I was using on Fedora 8 /
CCRMA. There were too many libraries and apps that were way too out of date
for my tastes, so I abandoned the whole experiment.
I might as well collate these questions here, for
what good it may do.
The kernel I'm configuring is 2.6.29.6 with matching RT patches (-rt23),
though all of the options listed here are actually present even in the
unpatched 2.6.29.6 kernel.
PREEMPT_RCU :
This was the one my original question was about in the earlier post.
In the newer kernels (even mainline ones, without any RT patches), there
is a choice of "RCU Subsystem", with one option being "classic", and
other being "preemptable". The choice would seem obvious for low
latency, except that the help text warns that a preemptible RCU is
likely to expose serious kernel bugs that may render the system
completely unstable. (This is *not* the same setting as the various
CONFIG_PREEMPT options that have been present in the mainline kernel for
awhile now. This is something new, and it appears in menuconfig as a
separate setting.) I think I've mostly gathered from reading through
the whole process of debugging that the kernel gurus went through that
as of last year or so, this is now considered mostly stable, after
having been subjected to a utility called 'rcutorture', and having fixed
many lockups. I'd still be interested in anything anyone else can
contribute about it though, especially if they're using it and they
also think it is stable.
There was a time when PREEMPT_RCU was unstable, and most of the -rt kernel
releases would fix one RCU bug while adding another. These times are now past,
however. PREEMPT_RCU has been completely stable for me on AMD and Intel (both
multicore) for about a year now.
GROUP_SCHED :
This one is interesting, and I don't know what to make of it, other than
that the LKML seems to have decided in the last two months or so that it
slows your system down and makes latencies worse.
The thing that's confusing about it though is that it's
described as a mechanism for grouping high priority tasks by group.
It's implied (though not spelled out specifically) that they even mean
by this Posix groups, because in one document I read, it says that
enabling this will cause you to be unable to get realtime as a non-root
user unless you are setup in a group specified in limits.conf. Hmm!
That sounds an awful lot like what we've just been calling pam_limits
for years now. Are we doing this with a kernel config option now? (One
which apparently doesn't work?)
The 3.0 (beta) version of 64Studio turns this on. Then again,
looking at their kernel config, they seem to turn everything on, and
that might be why it's considered beta. (The 2.1 stable version of
64Studio seems to have a kernel old enough that it never had that
option as a choice.) Anyone have any feedback about this? It's only in
the past two months that the kernel gurus have decided that it's bad,
but they're actually considering marking it broken from what I've read.
Here's another vote for completely broken! Every time I set up a box with the
-rt kernel, I start with the distro's .config and tweak from there. And every
time, I can't make realtime low-latency audio a reality until I disable
GROUP_SCHED.
CGROUPS :
This sounds cool, but I'm reasonably sure it's not actually necessary.
Documentation I've read suggests that this allows for letting
applications (or users) define CPU and memory pool affinity for tasks,
so that one could arbitrarily tie down particular threads or tasks to a
given processor core, or region of memory (or something like that).
However, the thing that makes me somewhat sure I don't positively need
this is that the same documentation also says that if you want to use
it, you need to create a new subdirectory under /dev, mount a new
pseudofilesystem under it, and then this module will populate that space
with dynamic configuration data about these affinity groups for running
tasks. I have neither seen any distro (even ones made for musicians)
that has set any such thing up out of the box, nor have I ever seen a
realtime howto that tells people to do it themselves. It sounds like
there's a lot of infrastructure necessary for this that's not common in
distros yet. (Though I see that regular Debian "lenny" turns this on in
the kernel without actually providing the special /dev support for it,
which I presume they're thinking you'll setup yourself if you're
interested.) Still, comments welcome...
As far as I can tell, CGROUPS doesn't affect realtime latencies enough to make
any sort of difference for realtime audio. It does, however, mess with the
scheduler, even without setting up the affinity groups. I've noted some
strange scheduling behavior: When running multiple instances of the same CPU
bound task (like burnP6, mprime, dd, or your favorite stress tester), the tasks
usually start off fairly evenly distributed between the cores, and then the
affinity seems to drift, lumping these tasks on one half of the cores while
leaving the other half mostly idle. On a core2quad with CGROUPS enabled, I
generally have to run 12 instances of burnP6 to keep utilization of all four
cores pegged at 99-100%. I wouldn't go as far as to say that CGROUPS is
broken, but I fail to see the usefulness on a production audio system with the
IRQ and other realtime priorities tuned properly, especially now that the core
scheduling features for -rt are stable.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
I've also run into a handful of config options that either help or hurt
realtime performance, depending on the specific hardware:
NO_HZ (seems to work properly on most hardware these days) PCI_MSI (some PCI devices seem to add more latency
with MSI)
RTC_DRV_CMOS (this depends on your particluar RTC hardware)
HPET_EMULATE_RTC (this depends on your particluar HPET hardware)
X86_ACPI_CPUFREQ (creates TSC drift between cores on Intel)
And of course, there's all kinds of profiling, stats, testing, and debugging
options that make any realtime load suffer. Occasionally, you'll find a device
driver that's just plain bad for -rt performance, but I've been running into
this problem a lot less these days.
Equally as important for low-latency are the scheduling priorities. After a lot
of google, lkml, and trial and error, I've settled on the following for
rock-solid low-latency audio:
99 FF migration
99 FF posixcputmr
98 FF IRQ-8 (realtime clock)
97 FF audio IRQ (in some cases means ieee1394 or specific USB port)
80 RR JACK
All audio and MIDI apps should run at a lower realtime priority than JACK. All
other IRQ- and sirq- threads should be set to
SCHED_OTHER. To set this up, I've added the following to my /etc/rc.local:
---
# set all irq threads to sched_other
for irq in `pgrep 'IRQ-'`; do
chrt -o -p 0 $irq;
done
# set all softirq threads to sched_other
for sirq in `pgrep 'sirq-'`; do
chrt -o -p 0 $sirq;
done
# set high prio IRQs
chrt -f -p 98 `pgrep IRQ-8` # rtc
chrt -f -p 97 `pgrep IRQ-16` # hda-intel
---
And of course, there's those pesky BIOS options. Some of the newer Intel CPU
features like to generate interrupts that the kernel can do absolutely nothing
about. I had to disable C1E, CPU TM function, execute disable bit, and one of
the ACPI options (can't remember which one) to get my core2quad to work.
Disabling video interrupts (only an issue on older hardware) always seems to
help. AMD systems seem to require much less (if any) BIOS tweaking for -rt.
Please, anyone, correct me if I'm wrong on any of this. Most of this knowledge
is gained from four years of following -rt development and tuning -rt kernels
on a variety of hardware. I'm always adding to my bag of tricks when it comes
to tuning realtime machines ;-}
Cheers,
--ww
_______________________________________________
Linux-audio-user mailing list
Linux-audio-user(a)lists.linuxaudio.org
http://lists.linuxaudio.org/mailman/listinfo/linux-audio-user
10:41 p.m.
New subject: More recent kernel config options
On Sun, 26 Jul 2009, Ken Restivo wrote:
Oddly (and I don't really understand this), you can have *both*. I've
seen recommendations online from people who claim to have gotten good
results with Jack even on laptops by having NO_HZ and HZ_1000 selected.
I don't really know what the effect of this is. Does this mean you
generate no ticks except when you need them, but when you do, the jiffy
rate is 1000Hz? This naively seems to me like a better idea than just
running a clock all the time, but I don't know how that pans out for
recording and MIDI.
It seems like there ought to be more pooling of resources regarding
complete kernel configs that work. Everyone always talks about the
patches themselves...but anyone who can do 'patch -p1' against a tree of
source code can get that part right.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
I've also
run into a handful of config options that either help or hurt
realtime performance, depending on the specific hardware:
NO_HZ (seems to work properly on most hardware these days)
Is that advised? I was told to use HZ_1000 in order to get MIDI to
work correctly. Should I be using NO_HZ instead? I've got several
machines to configure: a 32-bit Atom EEE, a 64-bit Intel Micro-ITX,
and a 64-bit Asus Intel Core2Duo
11:54 p.m.
New subject: More recent kernel config options
On Sun, 26 Jul 2009, Brent Busby wrote:
On Sun, 26 Jul 2009, Ken Restivo wrote:
Oddly (and I don't really understand this), you can have *both*. I've seen
recommendations online from people who claim to have gotten good results with
Jack even on laptops by having NO_HZ and HZ_1000 selected. I don't really
know what the effect of this is. Does this mean you generate no ticks except
when you need them, but when you do, the jiffy rate is 1000Hz? This naively
seems to me like a better idea than just running a clock all the time, but I
don't know how that pans out for recording and MIDI.
Ahh. This explains why I've had good results with NO_HZ for the
last year. I have both NO_HZ and HZ_1000 enabled. I still don't
understand how these two interact. At the very least, this would
provide the fallback for booting the kernel with nohz=off.
I've
also run into a handful of config options that either help or hurt
realtime performance, depending on the specific hardware:
NO_HZ (seems to work properly on most hardware these days)
Is that advised? I was told to use HZ_1000 in order to get MIDI to work
correctly. Should I be using NO_HZ instead? I've got several machines to
configure: a 32-bit Atom EEE, a 64-bit Intel Micro-ITX, and a 64-bit Asus
Intel Core2Duo It seems like there ought to be more pooling of
resources regarding complete
kernel configs that work. Everyone always talks about the patches
themselves...but anyone who can do 'patch -p1' against a tree of source code
can get that part right.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
I'm willing to post my config for the AMD laptop once I get the
right combination of HZ, RTC, and HPET options set. I'm still
running Fedora 8 on my Core2Quad, so I've stopped at 2.6.24.7-rt27
with everything running perfectly, but I can post the config from
that box if anyone is interested.
Cheers,
--ww
11:44 p.m.
New subject: More recent kernel config options
On Sun, 26 Jul 2009, Ken Restivo wrote:
On Sun, Jul 26, 2009 at 02:24:25PM -0700, William
Weston wrote:
I think this depends on where the MIDI sequencer gets it timing
from. On my Intel Core2Quad, NO_HZ is enabled and MIDI works fine.
I've been using MusE, BTW, which first tries to use the RTC, and
then falls back to the ALSA timer. On my AMD Turion, NO_HZ worked
fine with the legacy RTC driver, but MIDI timing completely fell
apart with the newer RTC_DRV_CMOS, and turning NO_HZ off didn't make
it any better.
I know that at one point, it was advised to keep NO_HZ turned off,
but it appears to work on at least some hardware now. Until I hear
a definitive answer that reflects the status of the current -rt
kernel tree, I'll continue to trust the trial and error method for
this one.
Alas, I can't test MIDI timing on MusE right now. After my upgrade
to Fedora 11, MusE won't even start up now. It appears to be a
futex hang during the memory pool allocation (before any new threads
are initialized), and I'm still trying to determine whether the
fault is in the kernel, the new glibc, or the new libstdc++. When I
have some spare time, I'll get all the debugging info together for
the dev list.
On Sat, 25 Jul 2009, Brent Busby wrote:
Is that advised? I was told to use HZ_1000 in order to get MIDI
to work correctly. Should I be using NO_HZ instead? I've got
several machines to configure: a 32-bit Atom EEE, a 64-bit Intel
Micro-ITX, and a 64-bit Asus Intel Core2Duo In addition to the question I posted earlier
about PREEMPT_RCU, I've
found still other kernel config options that are not covered in most of
the extant howtos for setting up low latency kernels, because they are
recently added in the unpatched kernel. (Or at least, they're more
recent than most of the howtos...)
I've been researching these options, mostly by googling (and
googling...and googling) the Linux Kernel Mailing List archives, and
also by looking at the config used by 64Studio, since it seems the
prevalent opinion is that they are stable as a rock. With the latter
approach of checking against 64Studio's config, I've had little luck
though, because their distro currently comes in a 2.1 "stable" version
that is based on Debian "etch", with a 2.6.21 kernel that predates the
existence of some of the config options I'm asking about; and a 3.0
"beta" version that uses a newer kernel, but should also probably be
taken with a grain of salt, because 64Studio is still working the bugs
out of it, and because it seems to have options turned on that the LKML
archives have warned strongly against. So, most of the information I've
gleaned has been from the LKML.
I tried out the 64Studio stable version last fall, and had a lot of troubles
with it. The 2.6.21 based kernel wouldn't even boot on my box, so I ended up
booting up with the 2.6.24 based -rt kernel that I was using on Fedora 8 /
CCRMA. There were too many libraries and apps that were way too out of date
for my tastes, so I abandoned the whole experiment.
I might as well collate these questions here, for
what good it may do.
The kernel I'm configuring is 2.6.29.6 with matching RT patches (-rt23),
though all of the options listed here are actually present even in the
unpatched 2.6.29.6 kernel.
PREEMPT_RCU :
This was the one my original question was about in the earlier post.
In the newer kernels (even mainline ones, without any RT patches), there
is a choice of "RCU Subsystem", with one option being "classic", and
other being "preemptable". The choice would seem obvious for low
latency, except that the help text warns that a preemptible RCU is
likely to expose serious kernel bugs that may render the system
completely unstable. (This is *not* the same setting as the various
CONFIG_PREEMPT options that have been present in the mainline kernel for
awhile now. This is something new, and it appears in menuconfig as a
separate setting.) I think I've mostly gathered from reading through
the whole process of debugging that the kernel gurus went through that
as of last year or so, this is now considered mostly stable, after
having been subjected to a utility called 'rcutorture', and having fixed
many lockups. I'd still be interested in anything anyone else can
contribute about it though, especially if they're using it and they
also think it is stable.
There was a time when PREEMPT_RCU was unstable, and most of the -rt kernel
releases would fix one RCU bug while adding another. These times are now past,
however. PREEMPT_RCU has been completely stable for me on AMD and Intel (both
multicore) for about a year now.
GROUP_SCHED :
This one is interesting, and I don't know what to make of it, other than
that the LKML seems to have decided in the last two months or so that it
slows your system down and makes latencies worse.
The thing that's confusing about it though is that it's
described as a mechanism for grouping high priority tasks by group.
It's implied (though not spelled out specifically) that they even mean
by this Posix groups, because in one document I read, it says that
enabling this will cause you to be unable to get realtime as a non-root
user unless you are setup in a group specified in limits.conf. Hmm!
That sounds an awful lot like what we've just been calling pam_limits
for years now. Are we doing this with a kernel config option now? (One
which apparently doesn't work?)
The 3.0 (beta) version of 64Studio turns this on. Then again,
looking at their kernel config, they seem to turn everything on, and
that might be why it's considered beta. (The 2.1 stable version of
64Studio seems to have a kernel old enough that it never had that
option as a choice.) Anyone have any feedback about this? It's only in
the past two months that the kernel gurus have decided that it's bad,
but they're actually considering marking it broken from what I've read.
Here's another vote for completely broken! Every time I set up a box with the
-rt kernel, I start with the distro's .config and tweak from there. And every
time, I can't make realtime low-latency audio a reality until I disable
GROUP_SCHED.
CGROUPS :
This sounds cool, but I'm reasonably sure it's not actually necessary.
Documentation I've read suggests that this allows for letting
applications (or users) define CPU and memory pool affinity for tasks,
so that one could arbitrarily tie down particular threads or tasks to a
given processor core, or region of memory (or something like that).
However, the thing that makes me somewhat sure I don't positively need
this is that the same documentation also says that if you want to use
it, you need to create a new subdirectory under /dev, mount a new
pseudofilesystem under it, and then this module will populate that space
with dynamic configuration data about these affinity groups for running
tasks. I have neither seen any distro (even ones made for musicians)
that has set any such thing up out of the box, nor have I ever seen a
realtime howto that tells people to do it themselves. It sounds like
there's a lot of infrastructure necessary for this that's not common in
distros yet. (Though I see that regular Debian "lenny" turns this on in
the kernel without actually providing the special /dev support for it,
which I presume they're thinking you'll setup yourself if you're
interested.) Still, comments welcome...
As far as I can tell, CGROUPS doesn't affect realtime latencies enough to make
any sort of difference for realtime audio. It does, however, mess with the
scheduler, even without setting up the affinity groups. I've noted some
strange scheduling behavior: When running multiple instances of the same CPU
bound task (like burnP6, mprime, dd, or your favorite stress tester), the tasks
usually start off fairly evenly distributed between the cores, and then the
affinity seems to drift, lumping these tasks on one half of the cores while
leaving the other half mostly idle. On a core2quad with CGROUPS enabled, I
generally have to run 12 instances of burnP6 to keep utilization of all four
cores pegged at 99-100%. I wouldn't go as far as to say that CGROUPS is
broken, but I fail to see the usefulness on a production audio system with the
IRQ and other realtime priorities tuned properly, especially now that the core
scheduling features for -rt are stable.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
I've also run into a handful of config options that either help or hurt
realtime performance, depending on the specific hardware:
NO_HZ (seems to work properly on most hardware these days) PCI_MSI (some
PCI devices seem to add more latency with MSI)
RTC_DRV_CMOS (this depends on your particluar RTC hardware)
HPET_EMULATE_RTC (this depends on your particluar HPET hardware)
X86_ACPI_CPUFREQ (creates TSC drift between cores on Intel)
And of course, there's all kinds of profiling, stats, testing, and debugging
options that make any realtime load suffer. Occasionally, you'll find a device
driver that's just plain bad for -rt performance, but I've been running into
this problem a lot less these days.
Equally as important for low-latency are the scheduling priorities. After a lot
of google, lkml, and trial and error, I've settled on the following for
rock-solid low-latency audio:
99 FF migration
99 FF posixcputmr
98 FF IRQ-8 (realtime clock)
97 FF audio IRQ (in some cases means ieee1394 or specific USB port)
80 RR JACK
All audio and MIDI apps should run at a lower realtime priority than JACK. All
other IRQ- and sirq- threads should be set to
SCHED_OTHER. To set this up, I've added the following to my /etc/rc.local:
---
# set all irq threads to sched_other
for irq in `pgrep 'IRQ-'`; do
chrt -o -p 0 $irq;
done
# set all softirq threads to sched_other
for sirq in `pgrep 'sirq-'`; do
chrt -o -p 0 $sirq;
done
# set high prio IRQs
chrt -f -p 98 `pgrep IRQ-8` # rtc
chrt -f -p 97 `pgrep IRQ-16` # hda-intel
---
And of course, there's those pesky BIOS options. Some of the newer Intel CPU
features like to generate interrupts that the kernel can do absolutely nothing
about. I had to disable C1E, CPU TM function, execute disable bit, and one of
the ACPI options (can't remember which one) to get my core2quad to work.
Disabling video interrupts (only an issue on older hardware) always seems to
help. AMD systems seem to require much less (if any) BIOS tweaking for -rt.
Please, anyone, correct me if I'm wrong on any of this. Most of this knowledge
is gained from four years of following -rt development and tuning -rt kernels
on a variety of hardware. I'm always adding to my bag of tricks when it comes
to tuning realtime machines ;-}
Cheers,
--ww
_______________________________________________
Linux-audio-user mailing list
Linux-audio-user(a)lists.linuxaudio.org
http://lists.linuxaudio.org/mailman/listinfo/linux-audio-user
5668
days inactive
5668
days old
linux-audio-user@lists.linuxaudio.org
4 comments
3 participants
participants (3)
-
Brent Busby -
Ken Restivo -
William Weston