[LAU] More recent kernel config options

Sun Jul 26 20:03:37 EDT 2009

On Sun, Jul 26, 2009 at 02:24:25PM -0700, William Weston wrote:
> On Sat, 25 Jul 2009, Brent Busby wrote:
> 
> > In addition to the question I posted earlier about PREEMPT_RCU, I've
> > found still other kernel config options that are not covered in most of
> > the extant howtos for setting up low latency kernels, because they are
> > recently added in the unpatched kernel.  (Or at least, they're more
> > recent than most of the howtos...)
> > 
> > I've been researching these options, mostly by googling (and
> > googling...and googling) the Linux Kernel Mailing List archives, and
> > also by looking at the config used by 64Studio, since it seems the
> > prevalent opinion is that they are stable as a rock.  With the latter
> > approach of checking against 64Studio's config, I've had little luck
> > though, because their distro currently comes in a 2.1 "stable" version
> > that is based on Debian "etch", with a 2.6.21 kernel that predates the
> > existence of some of the config options I'm asking about; and a 3.0
> > "beta" version that uses a newer kernel, but should also probably be
> > taken with a grain of salt, because 64Studio is still working the bugs
> > out of it, and because it seems to have options turned on that the LKML
> > archives have warned strongly against.  So, most of the information I've
> > gleaned has been from the LKML.
> 
> I tried out the 64Studio stable version last fall, and had a lot of troubles 
> with it.  The 2.6.21 based kernel wouldn't even boot on my box, so I ended up 
> booting up with the 2.6.24 based -rt kernel that I was using on Fedora 8 / 
> CCRMA.  There were too many libraries and apps that were way too out of date 
> for my tastes, so I abandoned the whole experiment.
> 
> > I might as well collate these questions here, for what good it may do.
> > The kernel I'm configuring is 2.6.29.6 with matching RT patches (-rt23),
> > though all of the options listed here are actually present even in the
> > unpatched 2.6.29.6 kernel.
> > 
> > 
> > 
> > PREEMPT_RCU :
> > 
> > This was the one my original question was about in the earlier post.
> > In the newer kernels (even mainline ones, without any RT patches), there
> > is a choice of "RCU Subsystem", with one option being "classic", and
> > other being "preemptable".  The choice would seem obvious for low
> > latency, except that the help text warns that a preemptible RCU is
> > likely to expose serious kernel bugs that may render the system
> > completely unstable.  (This is *not* the same setting as the various
> > CONFIG_PREEMPT options that have been present in the mainline kernel for
> > awhile now.  This is something new, and it appears in menuconfig as a
> > separate setting.)  I think I've mostly gathered from reading through
> > the whole process of debugging that the kernel gurus went through that
> > as of last year or so, this is now considered mostly stable, after
> > having been subjected to a utility called 'rcutorture', and having fixed
> > many lockups.  I'd still be interested in anything anyone else can
> > contribute about it though, especially if they're using it and they
> > also think it is stable.
> 
> There was a time when PREEMPT_RCU was unstable, and most of the -rt kernel 
> releases would fix one RCU bug while adding another.  These times are now past, 
> however.  PREEMPT_RCU has been completely stable for me on AMD and Intel (both 
> multicore) for about a year now.
> 
> > GROUP_SCHED :
> > 
> > This one is interesting, and I don't know what to make of it, other than
> > that the LKML seems to have decided in the last two months or so that it
> > slows your system down and makes latencies worse.
> > 	The thing that's confusing about it though is that it's
> > described as a mechanism for grouping high priority tasks by group.
> > It's implied (though not spelled out specifically) that they even mean
> > by this Posix groups, because in one document I read, it says that
> > enabling this will cause you to be unable to get realtime as a non-root
> > user unless you are setup in a group specified in limits.conf.  Hmm!
> > That sounds an awful lot like what we've just been calling pam_limits
> > for years now.  Are we doing this with a kernel config option now?  (One
> > which apparently doesn't work?)
> > 	The 3.0 (beta) version of 64Studio turns this on.  Then again,
> > looking at their kernel config, they seem to turn everything on, and
> > that might be why it's considered beta.  (The 2.1 stable version of
> > 64Studio seems to have a kernel old enough that it never had that
> > option as a choice.)  Anyone have any feedback about this?  It's only in
> > the past two months that the kernel gurus have decided that it's bad,
> > but they're actually considering marking it broken from what I've read.
> 
> Here's another vote for completely broken!  Every time I set up a box with the 
> -rt kernel, I start with the distro's .config and tweak from there.  And every 
> time, I can't make realtime low-latency audio a reality until I disable 
> GROUP_SCHED.
> 
> > CGROUPS :
> > 
> > This sounds cool, but I'm reasonably sure it's not actually necessary.
> > Documentation I've read suggests that this allows for letting
> > applications (or users) define CPU and memory pool affinity for tasks,
> > so that one could arbitrarily tie down particular threads or tasks to a
> > given processor core, or region of memory (or something like that).
> > However, the thing that makes me somewhat sure I don't positively need
> > this is that the same documentation also says that if you want to use
> > it, you need to create a new subdirectory under /dev, mount a new
> > pseudofilesystem under it, and then this module will populate that space
> > with dynamic configuration data about these affinity groups for running
> > tasks.  I have neither seen any distro (even ones made for musicians)
> > that has set any such thing up out of the box, nor have I ever seen a
> > realtime howto that tells people to do it themselves.  It sounds like
> > there's a lot of infrastructure necessary for this that's not common in
> > distros yet.  (Though I see that regular Debian "lenny" turns this on in
> > the kernel without actually providing the special /dev support for it,
> > which I presume they're thinking you'll setup yourself if you're
> > interested.)  Still, comments welcome...
> 
> As far as I can tell, CGROUPS doesn't affect realtime latencies enough to make 
> any sort of difference for realtime audio.  It does, however, mess with the 
> scheduler, even without setting up the affinity groups.  I've noted some 
> strange scheduling behavior: When running multiple instances of the same CPU 
> bound task (like burnP6, mprime, dd, or your favorite stress tester), the tasks 
> usually start off fairly evenly distributed between the cores, and then the 
> affinity seems to drift, lumping these tasks on one half of the cores while 
> leaving the other half mostly idle.  On a core2quad with CGROUPS enabled, I 
> generally have to run 12 instances of burnP6 to keep utilization of all four 
> cores pegged at 99-100%.  I wouldn't go as far as to say that CGROUPS is 
> broken, but I fail to see the usefulness on a production audio system with the 
> IRQ and other realtime priorities tuned properly, especially now that the core 
> scheduling features for -rt are stable.
> 
> > -- 
> > + Brent A. Busby	 + "We've all heard that a million monkeys
> > + UNIX Systems Admin	 +  banging on a million typewriters will
> > + University of Chicago	 +  eventually reproduce the entire works of
> > + Physical Sciences Div. +  Shakespeare.  Now, thanks to the Internet,
> > + James Franck Institute +  we know this is not true." -Robert Wilensky
> 
> I've also run into a handful of config options that either help or hurt 
> realtime performance, depending on the specific hardware:
> 
> NO_HZ (seems to work properly on most hardware these days)

Is that advised? I was told to use HZ_1000 in order to get MIDI to work correctly. Should I be using NO_HZ instead? I've got several machines to configure: a 32-bit Atom EEE, a  64-bit Intel Micro-ITX, and a 64-bit Asus Intel Core2Duo

> PCI_MSI (some PCI devices seem to add more latency with MSI)
> RTC_DRV_CMOS (this depends on your particluar RTC hardware)
> HPET_EMULATE_RTC (this depends on your particluar HPET hardware)
> X86_ACPI_CPUFREQ (creates TSC drift between cores on Intel)
> 
> And of course, there's all kinds of profiling, stats, testing, and debugging 
> options that make any realtime load suffer.  Occasionally, you'll find a device 
> driver that's just plain bad for -rt performance, but I've been running into 
> this problem a lot less these days.
> 
> Equally as important for low-latency are the scheduling priorities. After a lot 
> of google, lkml, and trial and error, I've settled on the following for 
> rock-solid low-latency audio:
> 
> 99 FF migration
> 99 FF posixcputmr
> 98 FF IRQ-8 (realtime clock)
> 97 FF audio IRQ (in some cases means ieee1394 or specific USB port)
> 80 RR JACK
> 
> All audio and MIDI apps should run at a lower realtime priority than JACK.  All 
> other IRQ- and sirq- threads should be set to
> SCHED_OTHER.  To set this up, I've added the following to my /etc/rc.local:
> 
> ---
> # set all irq threads to sched_other
> for irq in `pgrep 'IRQ-'`; do
>  	chrt -o -p 0 $irq;
> done
> 
> # set all softirq threads to sched_other
> for sirq in `pgrep 'sirq-'`; do
>  	chrt -o -p 0 $sirq;
> done
> 
> # set high prio IRQs
> chrt -f -p 98 `pgrep IRQ-8`	# rtc
> chrt -f -p 97 `pgrep IRQ-16`	# hda-intel
> ---
> 
> And of course, there's those pesky BIOS options.  Some of the newer Intel CPU 
> features like to generate interrupts that the kernel can do absolutely nothing 
> about.  I had to disable C1E, CPU TM function, execute disable bit, and one of 
> the ACPI options (can't remember which one) to get my core2quad to work. 
> Disabling video interrupts (only an issue on older hardware) always seems to 
> help.  AMD systems seem to require much less (if any) BIOS tweaking for -rt.
> 
> Please, anyone, correct me if I'm wrong on any of this.  Most of this knowledge 
> is gained from four years of following -rt development and tuning -rt kernels 
> on a variety of hardware.  I'm always adding to my bag of tricks when it comes 
> to tuning realtime machines ;-}
> 
> 
> Cheers,
> --ww
> _______________________________________________
> Linux-audio-user mailing list
> Linux-audio-user at lists.linuxaudio.org
> http://lists.linuxaudio.org/mailman/listinfo/linux-audio-user