19 Jul
2009
19 Jul
'09
2:18 p.m.
Looking at the various sources online giving advice about setting up
realtime pre-emption in /etc/security/limits.conf, I found this mildly
amusing list of recommendations from various sources. I've changed the
group to 'realtime' in all cases just because that's what I use.
# Gentoo:
* hard rtprio 0
* soft rtprio 0
@realtime hard rtprio 20
@realtime soft rtprio 10
# Gentoo (Pro-Audio Overlay):
@realtime - rtprio 90
@realtime - nice -5
@realtime - memlock 500000
# Arch:
@realtime - rtprio 70
@realtime - nice -10
@realtime - memlock 250000
# Suse:
@realtime - rtprio 100
@realtime - nice -10
@realtime - memlock 400000
# Debian:
@realtime - rtprio 99
@realtime - nice -15
@realtime - memlock 250000
# Ubuntu:
@realtime - memlock 0
@realtime - rtprio 99
# Jack:
@realtime - rtprio 99
@realtime - memlock unlimited
# Alsa:
@realtime - rtprio 95
@realtime - memlock 512000
@realtime - nice -19
Hey, with concensus like this, how can you go wrong! :)
These are the main questions this brought up for me:
- Hard and soft limits: Most of these examples don't set separate hard
and soft limits for anything, with the exception of the generic Gentoo
realtime docs at the top. Since the soft limit is defined in the
kernel.org documentation for pam_limits as "default values, for normal
system usage", does this mean that if my account is in a group that
sets a "soft" rtprio of 10 as shown in that example, that every
process I launch gets that setting? A program with such permission
still has to specifically request realtime, doesn't it, even with this
"default" soft limit being present? It makes it kind of meaningless
if your text editor, your desktop clock, and your file manager can all
get that priority too automatically just because it's a "default", or
so I'd guess anyway...
- For most of the other examples, the '-' symbol is being used to set
the hard and soft limits to be the same. Presumably this makes the
"default for normal system usage" as sky-high as the maximum, as in
the cases where a rtprio of 99 and unlimited memory locking is shown.
This makes me think (hope?) even more that this is something an app
needs to specifically request by asking the kernel for realtime, even
under an account that has the PAM permissions!
- Many sources actually say in their accompanying texts that allowing a
negative niceness is not necessary, and thus do not. Others set a
mild one ('-5' in the Gentoo Pro-Audio Overlay), and some like the
Alsa Project documentation recommend scary-looking ones like '-19'
that seem bound to get in the way of kernel services, if I'm not
misinterpreting. How "not nice" should we allow a recording app to
get? Or are the sources that don't set this on the right track?
- Memlock is fairly self-explanatory to me. I'd presume you want to
allow an app to memlock as much resident space as you expect your
largest pro-audio app to take. I was wondering if there was anything
to fear from non-pro-audio apps though in cases where the setting was
"unlimited". This seems a possible loss of stability should one of
your desktop programs go crazy for some reason...or is this, again,
something that a program would have to request via a special call to
the kernel, which wouldn't be likely from random errant behavior?
What I'd really like to be able to do is run Qsampler/LinuxSampler
with very large Gigasampler format libraries, and have it work well.
I have 8GB of memory on my system, so should I memlock say 4-6GB or
so?
The common factor in all these questions is:
Do these abilities have to be requested by a process through an
extraordinary call the kernel, or does this open your whole desktop up
to rampant memory usage, process priority elevation, etc. all the time?
I'd mainly like only my pro-audio apps to make use of these abilities,
even if I do grant them to my user account.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
19 Jul
19 Jul
8:15 p.m.
On Sun, Jul 19, 2009 at 12:18 PM, Brent Busby<brent(a)keycorner.org> wrote:
Looking at the various sources online giving advice
about setting up
realtime pre-emption in /etc/security/limits.conf, I found this mildly
amusing list of recommendations from various sources. I've changed the
group to 'realtime' in all cases just because that's what I use.
The nice value is irrelevant. Anybody attempting to use nice(2) to
improve realtime performance doesn't understand what they are doing.
It is very unfortunate that this has crept into so many limits.conf
examples.
This makes me think (hope?) even more that this is
something an app
needs to specifically request by asking the kernel for realtime, even
under an account that has the PAM permissions!
an app does not get RT scheduling without asking for it or being
granted it by another app that asks for it as a proxy.
no app uses memory locking without explicitly asking for it itself.
- Memlock is fairly self-explanatory to me. I'd
presume you want to
allow an app to memlock as much resident space as you expect your
largest pro-audio app to take. I was wondering if there was anything
to fear from non-pro-audio apps though in cases where the setting was
"unlimited".
"unlimited" is an easy answer. a better value would be something like
80% of physical RAM. unfortunately, PAM appears to have no way of
handling (or even parsing) such a specification, so the user or some
install script would have to determine it, and if they ever install or
remove RAM it would need manually fixing.
Do these abilities have to be requested by a process
through an
extraordinary call the kernel, or does this open your whole desktop up
to rampant memory usage, process priority elevation, etc. all the time?
I'd mainly like only my pro-audio apps to make use of these abilities,
even if I do grant them to my user account.
apps that do not current ask for SCHED_FIFO/SCHED_RR and/or do not ask
for memlock will not start using these facilities just because you've
made it possible for any process running as you to do so.
20 Jul
20 Jul
1:17 a.m.
On Mon, Jul 20, 2009 at 7:15 AM, Paul Davis <paul(a)linuxaudiosystems.com>wrote;wrote:
The nice value is irrelevant. Anybody attempting to use nice(2) to
improve realtime performance doesn't understand what they are doing.
It is very unfortunate that this has crept into so many limits.conf
examples.
Can you expand on this a little? Are you saying there is absolutely no need
to include a nice setting in limits.conf? Is this just some wacky tradition
that has been handed on between linux audio distros and metadistros? Don't
get me wrong, you probably know better than I, but it'd sure be *nice* to
hear an informed reason to abandon this practice.
10:21 a.m.
On Sun, Jul 19, 2009 at 11:17 PM, michael noble<looplog(a)gmail.com> wrote:
On Mon, Jul 20, 2009 at 7:15 AM, Paul Davis <paul(a)linuxaudiosystems.com>
wrote:
there are a few applications out there that seem to believe that you
can improve audio performance (i.e. less dropouts under load) by
changing their own nice value. this represents a complete
misunderstanding of the differences between conventional Unix
scheduling ("SCHED_OTHER") and realtime scheduling like SCHED_FIFO and
SCHED_RR. using nice can, under a few circumstances, make a
difference, but its use basically means that the developer(s) really
don't understand what the issues are.
presumably some distros believe that their version of limits.conf
should accomadate this kind of misconception. i don't think that
limits.conf should be specifying a nice value, at least not in
connection with audio/music applications that inherently need realtime
scheduling.
The nice value is irrelevant. Anybody attempting to use nice(2) to
improve realtime performance doesn't understand what they are doing.
It is very unfortunate that this has crept into so many limits.conf
examples.
Can you expand on this a little? Are you saying there is absolutely no need
to include a nice setting in limits.conf? Is this just some wacky tradition
that has been handed on between linux audio distros and metadistros? Don't
get me wrong, you probably know better than I, but it'd sure be nice to hear
an informed reason to abandon this practice. 
10:35 a.m.
On Mon, 20 Jul 2009, Paul Davis wrote:
presumably some distros believe that their version of
limits.conf
should accomadate this kind of misconception. i don't think that
limits.conf should be specifying a nice value, at least not in
connection with audio/music applications that inherently need realtime
scheduling.
limits.conf doesn't _specify_ a nice value. Instead, it limits the
range that the user can set (e.g. 'nice -n -10 emacs').
Most default installs won't let a normal user set a nice value below 0.
If you change limits.conf, they are capable of setting lower nice
values... but by default still starts with nice==0.
Peace,
Gabriel
11:29 a.m.
On Mon, Jul 20, 2009 at 7:35 AM, Gabriel M.
Beddingfield<gabriel(a)teuton.org> wrote:
On Mon, 20 Jul 2009, Paul Davis wrote:
I believe Paul meant "should not specify a lower than normal nice
limit". You were discussing limits.conf, after all. He knows that
does not change the default nice value of 0.
--
joq
presumably some distros believe that their
version of limits.conf
should accomadate this kind of misconception. i don't think that
limits.conf should be specifying a nice value, at least not in
connection with audio/music applications that inherently need realtime
scheduling.
limits.conf doesn't _specify_ a nice value. Instead, it limits the
range that the user can set (e.g. 'nice -n -10 emacs').
Most default installs won't let a normal user set a nice value below 0.
If you change limits.conf, they are capable of setting lower nice
values... but by default still starts with nice==0.
11:41 a.m.
On Mon, Jul 20, 2009 at 7:21 AM, Paul Davis<paul(a)linuxaudiosystems.com> wrote:
there are a few applications out there that seem to
believe that you
can improve audio performance (i.e. less dropouts under load) by
changing their own nice value. this represents a complete
misunderstanding of the differences between conventional Unix
scheduling ("SCHED_OTHER") and realtime scheduling like SCHED_FIFO and
SCHED_RR. using nice can, under a few circumstances, make a
difference, but its use basically means that the developer(s) really
don't understand what the issues are.
presumably some distros believe that their version of limits.conf
should accomadate this kind of misconception. i don't think that
limits.conf should be specifying a nice value, at least not in
connection with audio/music applications that inherently need realtime
scheduling.
Despite the fact that negative nice values are ineffective for
achieving solid realtime audio, I doubt we'll see many distributions
jumping into the role of discouraging that style of programming.
Most distribution developers see their role as packaging Linux
applications in a form that makes them easily accessible to end users.
They generally avoid highly technical discussions about "how those
applications should be written".
If enough users want to run "nice-audio" applications, they are likely
to enable that behavior. Why shouldn't they?
--
joq
1:33 p.m.
On Mon, Jul 20, 2009 at 9:41 AM, Jack O'Quin<jack.oquin(a)gmail.com> wrote:
Despite the fact that negative nice values are
ineffective for
achieving solid realtime audio, I doubt we'll see many distributions
jumping into the role of discouraging that style of programming.
Most distribution developers see their role as packaging Linux
applications in a form that makes them easily accessible to end users.
They generally avoid highly technical discussions about "how those
applications should be written".
If enough users want to run "nice-audio" applications, they are likely
to enable that behavior. Why shouldn't they?
given that many distributions have actively resisted enabling the
correct approach to writing such applications, i don't see why they
should not be encouraged to reverse themselves on both fronts: enable
the right way, and discourage the wrong way. it is crazy to claim that
they simply want to make things easily accessible to end users - the
debian packagers, for example, have argued that using SCHED_{FIFO,RR}
is wrong and that no app should be using memlock. so, they *do* take
positions ... i'm just saying they need a new one, and that is that
making lower nice values available for *this* purpose is wrong. there
may, of course, be other reasons to permit it.
--p
2:09 p.m.
On Mon, Jul 20, 2009 at 10:33 AM, Paul Davis<paul(a)linuxaudiosystems.com> wrote:
On Mon, Jul 20, 2009 at 9:41 AM, Jack
O'Quin<jack.oquin(a)gmail.com> wrote:
I see your point and share your frustration.
To be fair (not always easy), their objection to genuine realtime
scheduling rests on a perception (real or imagined) that these tools
may open up Denial of Service attacks on *all* users -- not just the
few doing serious audio production. Although I am not aware of any
genuine DoS attacks of that sort, distribution developers still have a
responsibility to take the possibility seriously.
So even here, I believe their motive remains to make things as easy as
possible for (most) users to do (most) everything they like. For a
general-purpose distribution, that's probably the best anyone can do.
Despite the fact that negative nice values are
ineffective for
achieving solid realtime audio, I doubt we'll see many distributions
jumping into the role of discouraging that style of programming.
Most distribution developers see their role as packaging Linux
applications in a form that makes them easily accessible to end users.
They generally avoid highly technical discussions about "how those
applications should be written".
If enough users want to run "nice-audio" applications, they are likely
to enable that behavior. Why shouldn't they?
given that many distributions have actively resisted enabling the
correct approach to writing such applications, i don't see why they
should not be encouraged to reverse themselves on both fronts: enable
the right way, and discourage the wrong way. it is crazy to claim that
they simply want to make things easily accessible to end users - the
debian packagers, for example, have argued that using SCHED_{FIFO,RR}
is wrong and that no app should be using memlock. so, they *do* take
positions ... i'm just saying they need a new one, and that is that
making lower nice values available for *this* purpose is wrong. there
may, of course, be other reasons to permit it. From their point of view, "nice -10" is quite
harmless, so they see no
need to disallow it.
--
joq
3:37 p.m.
On Mon, 20 Jul 2009, Paul Davis wrote:
the right way, and discourage the wrong way. it is
crazy to claim that
they simply want to make things easily accessible to end users - the
debian packagers, for example, have argued that using SCHED_{FIFO,RR}
is wrong and that no app should be using memlock. so, they *do* take
positions ... i'm just saying they need a new one, and that is that
making lower nice values available for *this* purpose is wrong. there
may, of course, be other reasons to permit it.
As much as Debian tries to be a general purpose OS, I've found that they
tend to be most at home thinking about multiuser servers and multiple
user Linux workstations, after the style of old Sun machines. They're
really good at that, but they're definitely not musicians.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
4:07 p.m.
On Mon, Jul 20, 2009 at 12:37 PM, Brent Busby<brent(a)keycorner.org> wrote:
As much as Debian tries to be a general purpose OS,
I've found that they
tend to be most at home thinking about multiuser servers and multiple
user Linux workstations, after the style of old Sun machines. They're
really good at that, but they're definitely not musicians.
Agreed.
If you want a Debian-based system, Ubuntu is much more focused on
requirements for a general-purpose desktop. Not specifically for
musicians of course, but easier to adapt for that purpose.
--
joq
29 Jul
29 Jul
4:20 a.m.
Jack O'Quin wrote:
On Mon, Jul 20, 2009 at 12:37 PM, Brent
Busby<brent(a)keycorner.org> wrote:
maybe it is more of a trade-off -
Ubuntu may be better suited to your hardware and usage, and may offer
the ways to configure the system how you like it - if so great, you are
lucky that all that has been set up for you!
- but you may get more flexibility at the cost of more initial
learning time and configuration effort using more general distributions
like Debian or Gentoo.
Sidux, for example, offers a more up-to-date and more desktop-oriented
starting point for a Debian system than the very careful and
conservative policies in Debian Stable, which certainly IS focussed on
the needs of big servers etc.
Plus you will need to add at least the Debian Multimedia repositories -
or the very strict legal and open source policies of Debian will make
life difficult.
I believe it is a good thing that Debian keeps its repositories very
strictly legal, and strictly GPL compatible, and has a very conservative
view of Stable, since adding extra repositories to match your personal
needs and beliefs is easy, and using Unstable is a fine option for
desktop use.
Simon
As much as Debian tries to be a general purpose
OS, I've found that they
tend to be most at home thinking about multiuser servers and multiple
user Linux workstations, after the style of old Sun machines. They're
really good at that, but they're definitely not musicians.
Agreed.
If you want a Debian-based system, Ubuntu is much more focused on
requirements for a general-purpose desktop. Not specifically for
musicians of course, but easier to adapt for that purpose.
20 Jul
20 Jul
2:59 p.m.
On Sun, 19 Jul 2009, Paul Davis wrote:
[...] details on pam_limits snipped
Thank you for this information... That answers all my questions.
--
+ Brent A. Busby + "We've all heard that a million monkeys
+ UNIX Systems Admin + banging on a million typewriters will
+ University of Chicago + eventually reproduce the entire works of
+ Physical Sciences Div. + Shakespeare. Now, thanks to the Internet,
+ James Franck Institute + we know this is not true." -Robert Wilensky
4:21 p.m.
Brent Busby kirjoitti:
On Sun, 19 Jul 2009, Paul Davis wrote:
[...] details on pam_limits snipped
Thank you for this information... That answers all my questions.
I got one more. I just installed new Eeebuntu version to my EEE and am
not able to get enough memory locked. Or something like that. I have
used the memlock 250000 setting in the past and it have worked but if I
try it now, Jack don't even start. If I lower the memlock setting lower,
at some point I get the Jack started, but memory amount is so low that I
got xruns like hell.
If I run Jack server + apps as sudo, everything works smoothly.
Does anyone have an idea how I could get everything working without
sudoing the apps?
Thanks,
.........jari.
21 Jul
21 Jul
8:22 a.m.
Quoting "Jari Suominen" <jari.suominen(a)helsinki.fi>fi>:
Brent Busby kirjoitti:
To diagnose the issue, we would need to see the exact error message.
Meanwhile, you could try checking whether the limits you've put in
limits.conf are actually in effect.
In a (bash) shell, running:
ulimit -l will tell you how much memory you can lock
ulimit -r will tell you at what realtime priority you can run software
On a system like a netbook, you can also try 'jackd --no-mlock' or
'jackd --unlock'. Unlock will make jack not lock some parts of the
memory (gui toolkits etc.), no-mlock just skips memlock totally.
Sampo
On Sun, 19 Jul 2009, Paul Davis wrote:
[...] details on pam_limits snipped
Thank you for this information... That answers all my questions.
I got one more. I just installed new Eeebuntu version to my EEE and am
not able to get enough memory locked. Or something like that. I have
used the memlock 250000 setting in the past and it have worked but if I
try it now, Jack don't even start. If I lower the memlock setting lower,
at some point I get the Jack started, but memory amount is so low that I
got xruns like hell.
10:52 a.m.
Sampo Savolainen kirjoitti:
Quoting "Jari Suominen"
<jari.suominen(a)helsinki.fi>fi>:
This is the messages printed in messages window of qjackctl:
15:41:20.664 Patchbay activated.
15:41:20.667 Statistics reset.
15:41:20.738 ALSA connection graph change.
15:41:20.929 ALSA active patchbay scan...
15:41:20.932 ALSA connection change.
15:41:21.136 ALSA active patchbay scan...
15:41:35.807 Startup script...
15:41:35.809 artsshell -q terminate
sh: artsshell: not found
15:41:36.230 Startup script terminated with exit status=32512.
15:41:36.232 JACK is starting...
15:41:36.234 /usr/bin/jackd -R -p128 -dalsa -dhw:0 -r48000 -p256 -n4
15:41:36.244 JACK was started with PID=3261.
no message buffer overruns
jackd 0.116.1
Copyright 2001-2005 Paul Davis and others.
jackd comes with ABSOLUTELY NO WARRANTY
This is free software, and you are welcome to redistribute it
under certain conditions; see the file COPYING for details
JACK compiled with System V SHM support.
loading driver ..
apparent rate = 48000
creating alsa driver ... hw:0|hw:0|256|4|48000|0|0|nomon|swmeter|-|32bit
control device hw:0
15:41:37.555 JACK was stopped successfully.
15:41:37.557 Post-shutdown script...
15:41:37.559 killall jackd
15:41:37.561 JACK has crashed.
jackd: no process killed
15:41:37.980 Post-shutdown script terminated with exit status=256.
15:41:38.442 Could not connect to JACK server as client. - Overall
operation failed. - Unable to connect to server. Please check the
messages window for more info.
It seems to me that the actual cause is not revealed in these messages,
perhaps there is better log somewhere else?
Brent Busby kirjoitti:
To diagnose the issue, we would need to see the exact error message.
Meanwhile, you could try checking whether the limits you've put in
limits.conf are actually in effect. On Sun, 19 Jul 2009, Paul Davis wrote:
[...] details on pam_limits snipped
Thank you for this information... That answers all my questions.
I got one more.
I just installed new Eeebuntu version to my EEE and am
not able to get enough memory locked. Or something like that. I have
used the memlock 250000 setting in the past and it have worked but if I
try it now, Jack don't even start. If I lower the memlock setting lower,
at some point I get the Jack started, but memory amount is so low that I
got xruns like hell. In a (bash) shell, running:
ulimit -l will tell you how much memory you can lock
ulimit -r will tell you at what realtime priority you can run software
These will give me right numbers that I have set in limits.conf (250000,99).
On a system like a netbook, you can also try
'jackd --no-mlock' or
'jackd --unlock'. Unlock will make jack not lock some parts of the
memory (gui toolkits etc.), no-mlock just skips memlock totally.
These don't seem to help much.
But the thing I'm also interested is why same Jack settings work when
running qjackctl as sudo?
.j
22 Jul
22 Jul
7:03 a.m.
Quoting "Jari Suominen" <jari.suominen(a)helsinki.fi>fi>:
Sampo Savolainen kirjoitti:
Those messages look entirely ok, except for jackd just crashing! Very
weird. Can you try launching jackd with the same parameters from the
command line? Maybe that will give you a bit more information.
It seems that something has gone awry with the installation / system.
Maybe some files jackd needs are owned by root and not writable for
the user?
You can always strace jackd:
strace -f /usr/bin/jackd [parameters] >& logfile
(I recommend not using realtime mode when stracing. But you might need
to do that if jackd crashes only when in realtime mode.)
The log will tell you all system operations the process does. At the
end, just before the crash, you should see some syscalls go awry. They
are most likely permission related as it works as root, so look for
open() calls in particular.
Sampo
To diagnose the issue, we would need to see the
exact error message.
Meanwhile, you could try checking whether the limits you've put in
limits.conf are actually in effect.
This is the messages printed in messages window of qjackctl:
-- snip --
But the thing I'm also interested is why same Jack settings work when
running qjackctl as sudo?
24 Jul
24 Jul
1:22 p.m.
Sampo Savolainen kirjoitti:
Quoting "Jari Suominen"
<jari.suominen(a)helsinki.fi>fi>:
Somehow the problem fixed itself. At first jack crashed at the command
line to a 'bus error'. But on next run it worked flawlessly. Wonder what
happened there? But it works!
But there is still more xruns than when run as sudo. And wonder if the
RT letters at qjackctl should blink?
.............j
Sampo Savolainen kirjoitti:
Those messages look entirely ok, except for jackd just crashing! Very
weird. Can you try launching jackd with the same parameters from the
command line? Maybe that will give you a bit more information. To diagnose the issue, we would need to see the
exact error message.
Meanwhile, you could try checking whether the limits you've put in
limits.conf are actually in effect.
This is the messages printed in messages
window of qjackctl:
-- snip --
But the thing I'm also interested is why same Jack settings work when
running qjackctl as sudo?
27 Jul
27 Jul
6:46 a.m.
On Fri, 2009-07-24 at 18:24 +0300, Jari Suominen wrote:
Somehow the problem fixed itself. At first jack
crashed at the command
line to a 'bus error'. But on next run it worked flawlessly. Wonder what
happened there? But it works!
But there is still more xruns than when run as sudo. And wonder if the
RT letters at qjackctl should blink?
Are you sure you're using the same settings for root and yourself? If
you run qjackctl with sudo, it's probably using the preferences set for
the root account as opposed to running as yourself with your
preferences.
Sampo
28 Jul
28 Jul
7:43 a.m.
Sampo Savolainen kirjoitti:
On Fri, 2009-07-24 at 18:24 +0300, Jari Suominen
wrote:
sudo seems to be having rt-rio set to 99 (as @audio has) and memlock
limit as 64 (!) which is 250000 for @audio. So it should have less
rights than my @audio group!
XRuns might have something to do with wifi, try to set it off to see if
it helps. My system works somewhat randomyl that it might work smoothyl
half and hour or more or so and then xruns start to roll once in two
seconds or something. But it is hard to figure what causes what.
Hmm, now jack also started misbehave again. Sometimes it just works and
other times not. As the jack crashed to this 'bus error' if it crashes,
it might have something to do with missing artsshell? Could it just be
that sound server is somehow buzy and jackd can't get control over it
occasionally?
Also is there any way of using the patch bay functionability outside the
qjackctl? That would be sweet, but didn't yet find answer.
.j
Somehow the problem fixed itself. At first jack
crashed at the command
line to a 'bus error'. But on next run it worked flawlessly. Wonder what
happened there? But it works!
But there is still more xruns than when run as sudo. And wonder if the
RT letters at qjackctl should blink?
Are you sure you're using the same settings for root and yourself? If
you run qjackctl with sudo, it's probably using the preferences set for
the root account as opposed to running as yourself with your
preferences.
5389
days inactive
5399
days old
linux-audio-user@lists.linuxaudio.org
19 comments
8 participants
participants (8)
-
Brent Busby -
Gabriel M. Beddingfield -
Jack O'Quin -
Jari Suominen -
michael noble -
Paul Davis -
Sampo Savolainen -
Simon Wise