1 Dec
2009
1 Dec
'09
6:10 p.m.
Hello all,
Updates of jconv, now called jconvolver, and the zita-convolver
library are available on the usual place:
<http://www.kokkinizita.net/linuxaudio/downloads>/
The new zita-convolver is API-compatible to the previous
release but not binary compatible. If you have any code
using it please recompile.
Release notes for jconvolver and fconvolver 0.8.4
-------------------------------------------------
Jconvolver is a real-time convolution engine. It
can execute up to a 64 by 64 convolution matrix
(i.e. 4096 simultaneous convolutions) as long as
your CPU(s) can handle the load. It is designed
to be efficient also for sparse (e.g. diagonal)
matrices. Unused matrix elements do not take
any CPU time.
In contrast to e.g. BruteFir, jconvolver uses
multiple partition sizes, small ones at the start
of the impulse response and progressively longer
ones for the rest. This allows it provide both
zero processing delay while still remaining
efficient in CPU use.
Fconvolver performs the same processing on files
instead of real-time.
For this relaese the package and the binaries
have been renamed to avoid a conflict with an
existing 'jconv' package (related to the Japanese
language).
Functionally this version is identical to jconv
0.8.0. See README.CONFIG for the syntax of the
config files. Since 0.8.0 it has been possible
to name and connect the Jack ports. The related
commands have been added to README.CONFIG.
Jconvolver will work with libzita-convolver 1.0.0,
but upgrading to 2.0.0 is recommended. Note that
if you update the library you have to recompile
this package.
NOTE: Jconvolver will normally not work correctly
when used with Jack's freewheeling mode, e.g. when
exporting in Ardour. It _could_ actually work when
used on a single processor system, but don't rely
on this. This release of jconvolver will print a
warning when freewheel mode is entered.
The 'mkwavex' utility from previous versions has
been replaced by 'makemulti' which offers some more
options.
Enjoy !
--
FA
Wie der Mond heute Nacht aussieht !
Ist es nicht ein seltsames Bild ?
2 Dec
2 Dec
2:59 a.m.
On Tue, Dec 01, 2009 at 11:06:51PM +0100, fons(a)kokkinizita.net wrote:
Hello all,
Updates of jconv, now called jconvolver, and the zita-convolver
library are available on the usual place:
<http://www.kokkinizita.net/linuxaudio/downloads>/
The new zita-convolver is API-compatible to the previous
release but not binary compatible. If you have any code
using it please recompile.
Release notes for jconvolver and fconvolver 0.8.4
-------------------------------------------------
Jconvolver is a real-time convolution engine. It
can execute up to a 64 by 64 convolution matrix
(i.e. 4096 simultaneous convolutions) as long as
your CPU(s) can handle the load. It is designed
to be efficient also for sparse (e.g. diagonal)
matrices. Unused matrix elements do not take
any CPU time.
In contrast to e.g. BruteFir, jconvolver uses
multiple partition sizes, small ones at the start
of the impulse response and progressively longer
ones for the rest. This allows it provide both
zero processing delay while still remaining
efficient in CPU use.
Fconvolver performs the same processing on files
instead of real-time.
For this relaese the package and the binaries
have been renamed to avoid a conflict with an
existing 'jconv' package (related to the Japanese
language).
Functionally this version is identical to jconv
0.8.0. See README.CONFIG for the syntax of the
config files. Since 0.8.0 it has been possible
to name and connect the Jack ports. The related
commands have been added to README.CONFIG.
Jconvolver will work with libzita-convolver 1.0.0,
but upgrading to 2.0.0 is recommended. Note that
if you update the library you have to recompile
this package.
NOTE: Jconvolver will normally not work correctly
when used with Jack's freewheeling mode, e.g. when
exporting in Ardour. It _could_ actually work when
used on a single processor system, but don't rely
on this. This release of jconvolver will print a
warning when freewheel mode is entered.
Really? A huge number of tracks I've published, were mixed in Ardour freewheeling
mode, using jconv for reverb. Worked fine for me, at least with the older versions.
The 'mkwavex' utility from previous versions has
been replaced by 'makemulti' which offers some more
options.
6:13 a.m.
On Tue, Dec 01, 2009 at 10:57:58PM -0800, Ken Restivo wrote:
Really? A huge number of tracks I've published,
were
mixed in Ardour freewheeling mode, using jconv for
reverb. Worked fine for me, at least with the older versions.
You've been lucky :-) This will probably still work with
the new version (maybe even 'better' as the result of a
bug fix), but it's not by design.
In theory zita-convolver could be modified quite easily
to work correclty in freewheeling even on MP systems.
The complicated part is switching between normal mode
and freewheeling 'on the fly' - I've not worked out
how to do that correctly.
But as said, on a single processor system it will work.
What is happening is this: zita convolver uses some
processing threads that effectively run with a period
that is a power-of-2 multiple of Jack's basic period.
They run on RT priorities just below the one of Jack's
process thread (-1 for for each doubling of the period).
The output of these threads must be ready when their
next period starts.
Zita-convolver checks this condition and will report
errors if these threads are too slow, but it will not
wait for them - you're not supposed to wait in Jack's
callback, and if your system does support the CPU load
for the given configuration they will be ready, even
with some safety margin.
When freewheeling, Jack's main process thread will be
non-RT, while these extra threads remain at RT. So
they will pre-empts Jack's thread and will appear to
be always ready on time. Except when you have 2 CPUs:
then Jack's thread will be allowed to continue even
if some of the others have not yet completed their
work.
The issue is complicated a bit more by the way these
threads are scheduled, it is *not*
(viem with monospaced font)
1 1 1 1 1 1 1 1 1 1
2 2 2 2 2
4 4 4
8 8
but
1 1 1 1 1 1 1 1 1 1
2 2 2 2 2
4 4 4
8
i.e. in each period at most one one them
is at its longer period boundary.
Ciao,
--
FA
Wie der Mond heute Nacht aussieht !
Ist es nicht ein seltsames Bild ?
6:22 a.m.
New subject: [LAD] Update of jconv
Le 2 déc. 2009 à 11:09, fons(a)kokkinizita.net a écrit :
On Tue, Dec 01, 2009 at 10:57:58PM -0800, Ken Restivo
wrote:
So does this mean than going into freewheeling should be handled in a special way: like
dropping RT mode for additional worker threads and so on (then when freewheeling stops,
you would have to setup RT mode again...)
An quite interesting use case. I also always wondering how Ardour (using an additional
thread for disk access) handles freewheeling on/off, regarding thread management.
We could even imagine that JACK could provide some support for this kind of requirement :
allowing to define additional worker threads, automatically handling priority change when
freewheeling on/off....
Really? A huge number of tracks I've
published, were
mixed in Ardour freewheeling mode, using jconv for
reverb. Worked fine for me, at least with the older versions.
You've been lucky :-) This will probably still work with
the new version (maybe even 'better' as the result of a
bug fix), but it's not by design.
In theory zita-convolver could be modified quite easily
to work correclty in freewheeling even on MP systems.
The complicated part is switching between normal mode
and freewheeling 'on the fly' - I've not worked out
how to do that correctly.
But as said, on a single processor system it will work.
What is happening is this: zita convolver uses some
processing threads that effectively run with a period
that is a power-of-2 multiple of Jack's basic period.
They run on RT priorities just below the one of Jack's
process thread (-1 for for each doubling of the period).
The output of these threads must be ready when their
next period starts.
Zita-convolver checks this condition and will report
errors if these threads are too slow, but it will not
wait for them - you're not supposed to wait in Jack's
callback, and if your system does support the CPU load
for the given configuration they will be ready, even
with some safety margin.
When freewheeling, Jack's main process thread will be
non-RT, while these extra threads remain at RT. So
they will pre-empts Jack's thread and will appear to
be always ready on time. Except when you have 2 CPUs:
then Jack's thread will be allowed to continue even
if some of the others have not yet completed their
work.
The issue is complicated a bit more by the way these
threads are scheduled, it is *not*
(viem with monospaced font)
1 1 1 1 1 1 1 1 1 1
2 2 2 2 2
4 4 4
8 8
but
1 1 1 1 1 1 1 1 1 1
2 2 2 2 2
4 4 4
8
i.e. in each period at most one one them
is at its longer period boundary.
Ciao,
Stéphane
8:09 a.m.
New subject: [LAD] Update of jconv
On Wed, Dec 2, 2009 at 5:16 AM, Stéphane Letz <letz(a)grame.fr> wrote:
So does this mean than going into freewheeling should
be handled in a special way: like dropping RT mode for additional worker threads and so
on (then when freewheeling stops, you would have to setup RT mode again...)
it does imply that, yes.
An quite interesting use case. I also always wondering
how Ardour (using an additional thread for disk access) handles freewheeling on/off,
regarding thread management.
ardour's disk i/o thread ("the butler") doesn't run with RT scheduling,
ever.
there is a MIDI I/O thread (at least in 2.X) that tries to run with
low RT priority (it doesn't care if it doesn't get it). i haven't come
up with a reason yet to want to reset the scheduling of this thread.
We could even imagine that JACK could provide some
support for this kind of requirement : allowing to define additional worker threads,
automatically handling priority change when freewheeling on/off....
i think we offer enough already by offering functions to
acquire+release RT scheduling. unless we don't export them ... i doubt
if applications with such architectures want JACK itself to try to
micromanage this level of detail. they already have to make their own
decisions about thread priorities.
8:41 a.m.
New subject: [LAD] Update of jconv
On Wed, Dec 02, 2009 at 11:16:30AM +0100, Stéphane Letz wrote:
So does this mean than going into freewheeling should
be
handled in a special way: like dropping RT mode for
additional worker threads and so on (then when freewheeling
stops, you would have to setup RT mode again...)
See my other post - the easiest way would be to make the
main thread wait for the others.
We could even imagine that JACK could provide some
support
for this kind of requirement : allowing to define additional
worker threads, automatically handling priority change when
freewheeling on/off....
It could but I don't think it would simplify matters much.
Supporting many parellel period sizes *would* complicate
Jack a lot.
Paul Davis:
i doubt if applications with such architectures want
JACK
itself to try to micromanage this level of detail. they
already have to make their own decisions about thread
priorities.
These priorities maybe need some discussion.
The relative period sizes used by zita-convolver are
either 1:2:8:32 or 1:4:16:64. The first step is either
2 or 4, the others are always 4, and the choice of the
first step is made to make the longest one (if used)
always 8192.
In 1.0.0 each step would decrease the priority by one
relative to Jack's thread. In 2.0.0 this is different:
each doubling of the period will decrease the priority
by one.
This change ensures that threads using the same period
will have the same priority, even if they are not in
the same client. This is absolutely necessary in order
to make two or more such clients work together smoothly
and have 'fair' scheduling. This should become some sort
of 'standard' if more apps start using such schemes.
Threads doing disk-IO probably don't fall in this scheme,
I suspect they always use much larger block sizes, have
plenty of buffering and can run SCHED_OTHER.
Ciao,
--
FA
Wie der Mond heute Nacht aussieht !
Ist es nicht ein seltsames Bild ?
9:19 a.m.
New subject: [LAD] Update of jconv
On Wed, Dec 2, 2009 at 7:37 AM, <fons(a)kokkinizita.net> wrote:
This change ensures that threads using the same
period
will have the same priority, even if they are not in
the same client. This is absolutely necessary in order
to make two or more such clients work together smoothly
and have 'fair' scheduling. This should become some sort
of 'standard' if more apps start using such schemes.
my understanding of SCHED_FIFO says that it doesn't really work this
way. if you want to use equal priorities to ensure fair scheduling
among RT threads, you want SCHED_RR so that the kernel scheduler can
intervene and ensure roughly equivalent scheduling. if you use
SCHED_FIFO then every thread of the same priority will run until it
voluntarily yields. is this what you mean by "fair" scheduling? it
seems to me that whether this works or not depends entirely on what
the threads are actualy doing (i.e. how long before they are
guaranteed to yield the processor).
10:44 a.m.
New subject: [LAD] Update of jconv
On Wed, Dec 02, 2009 at 08:19:06AM -0500, Paul Davis wrote:
On Wed, Dec 2, 2009 at 7:37 AM,
<fons(a)kokkinizita.net> wrote:
In the general case this would be correct, SCHED_FIFO
does not in any way ensure that threads get a 'fair
share of time'. But this is not the general case.
In cases such as the one discussed, the threads are
triggered at a divisor of Jack's period rate. Each
time they get a quantum of work, perform it, and
wait for the next trigger. They do not try to run
for as long as they can.
The threads do not compete on CPU time: the assumption
is always that there is enough CPU power to perform
all the work. If this is not the case the whole thing
fails anyway.
The 'competition' is on the order in which they are
allowed to run, and for that priority is the only
criterion.
The objective of the priority scheme is to allow all
threads to meet their deadline, not to give them equal
CPU shares. They don't need equal CPU shares anyway,
in most cases the bulk of the work is done by the
lowest priority ones.
An example should make this clear. Assuming the
period size is 256, running the 'Great Hall' reverb
in jconvolver, configured for a minimum partition
size of 256 (no processing delay) will result in the
following threads:
(you can use -v to see this)
prio = 0, offs = 0, parsize = 256, npar = 3
prio = -1, offs = 768, parsize = 512, npar = 6
prio = -3, offs = 3840, parsize = 2048, npar = 6
prio = -5, offs = 16128, parsize = 8192, npar = 13
The first one is Jack's process thread, this will do the
work for 3 partitions of size 256. The second one will do
6 partitions of size 512, it will be triggered every two
periods. The next one runs every 8 Jack periods, the last
one every 32.
The last three are required to have completed their work
when the next trigger arrives. Their output is actually
only required later, 3 periods after the start for the
size 512, 15 periods for the size 2048, and 31 for the
last. This gives the whole system some robustness it
would not have if everything was planned 'just in time'.
Note that the extra margin is small for small partition
sizes and quite big for the larger ones.
Now if the same reverb is configured to use a minumum
period size of 4096 (because the user can tolerate the
delay and wants less CPU load) this would look different:
prio = -4, offs = 0, parsize = 4096, npar = 2
prio = -5, offs = 8192, parsize = 8192, npar = 14
As you can see the priorities match the partition size.
In this case no work is done in Jack's thread.
Now assume that I would use the scheme of version 1.0.0.
Then the priorities would be
-1 512
-2 2048
-3 8192
and
-1 4096
-2 8192
respectively. This would mean that the size 4096 thread
of the second instance would have the same priority as
the size 512 thread of the first. The time required for
the size 4096 calculations could easily be longer than
the period of of the size 256 thread. This would result
in the size 256 thread being late every time it has to
wait for the other to complete, which could happen every
16 periods.
Matching priorities to period size removes this problem,
and ensure that the most urgent work will always be done
first.
Ciao,
--
FA
Wie der Mond heute Nacht aussieht !
Ist es nicht ein seltsames Bild ?
This change ensures that threads using the same
period
will have the same priority, even if they are not in
the same client. This is absolutely necessary in order
to make two or more such clients work together smoothly
and have 'fair' scheduling. This should become some sort
of 'standard' if more apps start using such schemes.
my understanding of SCHED_FIFO says that it doesn't really work this
way. if you want to use equal priorities to ensure fair scheduling
among RT threads, you want SCHED_RR so that the kernel scheduler can
intervene and ensure roughly equivalent scheduling. if you use
SCHED_FIFO then every thread of the same priority will run until it
voluntarily yields. is this what you mean by "fair" scheduling? it
seems to me that whether this works or not depends entirely on what
the threads are actualy doing (i.e. how long before they are
guaranteed to yield the processor).
4:40 p.m.
New subject: [LAD] Update of jconv
On Wed, Dec 02, 2009 at 03:41:05PM +0100, fons(a)kokkinizita.net wrote:
On Wed, Dec 02, 2009 at 08:19:06AM -0500, Paul Davis
wrote:
Does the "partition" parameter in the conf file affect this?
On the reverbs I've been using successfully with freewheeling mode, the
"partition" is set to 2048, which is very high compared to the setting in the
sample reverbs tthat come with jconv.
-ken
On Wed, Dec 2, 2009 at 7:37 AM,
<fons(a)kokkinizita.net> wrote:
In the general case this would be correct, SCHED_FIFO
does not in any way ensure that threads get a 'fair
share of time'. But this is not the general case.
In cases such as the one discussed, the threads are
triggered at a divisor of Jack's period rate. Each
time they get a quantum of work, perform it, and
wait for the next trigger. They do not try to run
for as long as they can.
The threads do not compete on CPU time: the assumption
is always that there is enough CPU power to perform
all the work. If this is not the case the whole thing
fails anyway.
The 'competition' is on the order in which they are
allowed to run, and for that priority is the only
criterion.
The objective of the priority scheme is to allow all
threads to meet their deadline, not to give them equal
CPU shares. They don't need equal CPU shares anyway,
in most cases the bulk of the work is done by the
lowest priority ones.
An example should make this clear. Assuming the
period size is 256, running the 'Great Hall' reverb
in jconvolver, configured for a minimum partition
size of 256 (no processing delay) will result in the
following threads:
(you can use -v to see this)
prio = 0, offs = 0, parsize = 256, npar = 3
prio = -1, offs = 768, parsize = 512, npar = 6
prio = -3, offs = 3840, parsize = 2048, npar = 6
prio = -5, offs = 16128, parsize = 8192, npar = 13
The first one is Jack's process thread, this will do the
work for 3 partitions of size 256. The second one will do
6 partitions of size 512, it will be triggered every two
periods. The next one runs every 8 Jack periods, the last
one every 32.
The last three are required to have completed their work
when the next trigger arrives. Their output is actually
only required later, 3 periods after the start for the
size 512, 15 periods for the size 2048, and 31 for the
last. This gives the whole system some robustness it
would not have if everything was planned 'just in time'.
Note that the extra margin is small for small partition
sizes and quite big for the larger ones.
Now if the same reverb is configured to use a minumum
period size of 4096 (because the user can tolerate the
delay and wants less CPU load) this would look different:
prio = -4, offs = 0, parsize = 4096, npar = 2
prio = -5, offs = 8192, parsize = 8192, npar = 14
As you can see the priorities match the partition size.
In this case no work is done in Jack's thread.
Now assume that I would use the scheme of version 1.0.0.
Then the priorities would be
-1 512
-2 2048
-3 8192
and
-1 4096
-2 8192
respectively. This would mean that the size 4096 thread
of the second instance would have the same priority as
the size 512 thread of the first. The time required for
the size 4096 calculations could easily be longer than
the period of of the size 256 thread. This would result
in the size 256 thread being late every time it has to
wait for the other to complete, which could happen every
16 periods.
Matching priorities to period size removes this problem,
and ensure that the most urgent work will always be done
first.
This change ensures that threads using the same
period
will have the same priority, even if they are not in
the same client. This is absolutely necessary in order
to make two or more such clients work together smoothly
and have 'fair' scheduling. This should become some sort
of 'standard' if more apps start using such schemes.
my understanding of SCHED_FIFO says that it doesn't really work this
way. if you want to use equal priorities to ensure fair scheduling
among RT threads, you want SCHED_RR so that the kernel scheduler can
intervene and ensure roughly equivalent scheduling. if you use
SCHED_FIFO then every thread of the same priority will run until it
voluntarily yields. is this what you mean by "fair" scheduling? it
seems to me that whether this works or not depends entirely on what
the threads are actualy doing (i.e. how long before they are
guaranteed to yield the processor).
4:58 p.m.
New subject: [LAD] Update of jconv
On Wed, Dec 02, 2009 at 12:39:27PM -0800, Ken Restivo wrote:
Does the "partition" parameter in the conf
file affect this?
Yes, it determines the minumum partition size that will be
used. The only difference between the two examples I posted
is the value of this parameter, 256 or 4096.
On the reverbs I've been using successfully with
freewheeling
mode, the "partition" is set to 2048, which is very high compared
to the setting in the sample reverbs tthat come with jconv.
Which means you will have significant delay. If the reverb is
use as an effect (rather than to recreate real room acoustics)
this could actually be beneficial.
On a single processor machine freewheeling should just work,
and not depend on this paramter.
Ciao,
--
FA
Wie der Mond heute Nacht aussieht !
Ist es nicht ein seltsames Bild ?
3:09 a.m.
New subject: [LAD] Update of jconv
Am Dienstag, den 01.12.2009, 23:06 +0100 schrieb fons(a)kokkinizita.net:
Hello all,
Updates of jconv, now called jconvolver, and the zita-convolver
library are available on the usual place:
<http://www.kokkinizita.net/linuxaudio/downloads>/
The new zita-convolver is API-compatible to the previous
release but not binary compatible. If you have any code
using it please recompile.
Release notes for jconvolver and fconvolver 0.8.4
-------------------------------------------------
Jconvolver is a real-time convolution engine. It
can execute up to a 64 by 64 convolution matrix
(i.e. 4096 simultaneous convolutions) as long as
your CPU(s) can handle the load. It is designed
to be efficient also for sparse (e.g. diagonal)
matrices. Unused matrix elements do not take
any CPU time.
In contrast to e.g. BruteFir, jconvolver uses
multiple partition sizes, small ones at the start
of the impulse response and progressively longer
ones for the rest. This allows it provide both
zero processing delay while still remaining
efficient in CPU use.
Fconvolver performs the same processing on files
instead of real-time.
For this relaese the package and the binaries
have been renamed to avoid a conflict with an
existing 'jconv' package (related to the Japanese
language).
Functionally this version is identical to jconv
0.8.0. See README.CONFIG for the syntax of the
config files. Since 0.8.0 it has been possible
to name and connect the Jack ports. The related
commands have been added to README.CONFIG.
Jconvolver will work with libzita-convolver 1.0.0,
but upgrading to 2.0.0 is recommended. Note that
if you update the library you have to recompile
this package.
NOTE: Jconvolver will normally not work correctly
when used with Jack's freewheeling mode, e.g. when
exporting in Ardour. It _could_ actually work when
used on a single processor system, but don't rely
on this. This release of jconvolver will print a
warning when freewheel mode is entered.
The 'mkwavex' utility from previous versions has
been replaced by 'makemulti' which offers some more
options.
Enjoy !
Hi Fons
Thanks for the update, build and run fine here on debian/sid,
I only need to modify the makefile (add DESTDIR and change path) to
build my deb packets.
hermann
5278
days inactive
5279
days old
linux-audio-user@lists.linuxaudio.org
10 comments
5 participants
participants (5)
-
fons＠kokkinizita.net -
hermann -
Ken Restivo -
Paul Davis -
Stéphane Letz