Michael Ost wrote:
  The higher priority threads in the system are:
 ...
 * IRQ8 (rtc) - FIFO/99 
Why does this interrupt get such a high priority?
(Not that it matters as I don't expect it to be used at all ...)
  Are there issues with memory mapping, that can block
other unrelated
 threads? 
Then you would have seen page faults.
  There do appear to be "involuntary context
switches" (as reported by
 getrusage) when the spikes happen. This makes it seem like the
 scheduler is interrupting our threads. But how do you figure out why
 that is happening?  [...]  All of the 5 processing threads are
 SCHED_RR/76. [...]  Are there just too danged many SCHED_RR threads
 fighting for two cores? 
RR means Round Robin, i.e., all threads with the same priority get
an equal amount of CPU without much delay, so the scheduler has to
switch between them quite often.  RR is intended for threads that must
make some progress continuously.  (See "man sched_setscheduler" and
especially "man sched_rr_get_interval".)
If you want to run a thread until it has finished for now, use
SCHED_FIFO.
I assume it is your intention that all those threads have equal priority
(which means to the kernel "don't care which one of them gets executed
first").
Regards,
Clemens