5 Oct
2010
5 Oct
'10
6:12 a.m.
I'm writing on behalf of a friend who's having trouble dealing with midi
latency in a soft synth (possibly yoshimi) ...
Given a jack period of N frames, the midi latency with the original code
effectively ranges from N frames to 2 * N frames, which I guess qualifies
it as jittery. So far my friend has tried a few things, but there's no
workable solution as yet.
What seemed most promising was to break the audio generation into smaller
blocks, applying pending midi events between blocks. Sadly, that drags the
creation and destruction of note objects into the realtime jack process
callback path. Latency improves, but the number of notes you can get
away with before it all falls in a heap is significantly reduced.
Getting the destruction of dead notes out of the realtime path is trivial,
not so the creation of new ones. Even with a pool of pre-allocated note
objects, it seems the amount of initialization code per note is still a
real limiting factor on how busy things can get before it all falls apart.
Such is life (for my "friend").
cheers, Cal
5 Oct
5 Oct
9:51 a.m.
On Tue, Oct 05, 2010 at 03:12:12PM +1100, cal wrote:
I'm writing on behalf of a friend who's having
trouble dealing with midi
latency in a soft synth (possibly yoshimi) ...
Latency? Or jitter?
Given a jack period of N frames, the midi latency with
the original code
effectively ranges from N frames to 2 * N frames, which I guess qualifies
it as jittery. So far my friend has tried a few things, but there's no
workable solution as yet.
Of course how bad this is depends on the value of N, but yeah.
What seemed most promising was to break the audio
generation into smaller
blocks, applying pending midi events between blocks.
Isn't this what JACK does by choosing a period size? Splitting up the
processing withing a period like this seems odd.
I'm assuming you're talking about ALSA MIDI, here not JACK MIDI - is that
correct?
Sadly, that drags the creation and destruction of note
objects into the
realtime jack process callback path. (...) Even with a pool of pre-allocated
note objects, it seems the amount of initialization code per note is still a
real limiting factor on how busy things can get before it all falls apart.
Of course allocation is forbidden in the process() callback, so that's off
limits anyway. Using pre-allocated notes, it sounds kind of unlikely to me
that initializing your note objects is the bottleneck. It seems more likely
to me that your way of getting the notes from the MIDI device (again - is this
ALSA midi? Can you share code/patches?) is not RT-safe. Have you tried running
it with http://repo.or.cz/w/jack_interposer.git ?
To 'solve' ALSA MIDI jitter for JACK, you'll have to receive ALSA MIDI
messages
in a seperate thread, and make sure you apply them at the right position in the
process buffer based on a timestamp telling you when exactly the note event was
recorded.
With JACK MIDI, JACK takes care of the recording MIDI events and timestamping
them in a sample-accurate way - all you'll have to do is make sure you do take
into account the timestamp to correctly place them in the process buffer.
This will only 'solve' jitter though: a latency proportional to the period size
is inherent to the way JACK works I'm afraid. Hence the desirability of tuning
your system to allow for low period sizes.
Regards,
Arnout
11:27 a.m.
On 05/10/10 18:51, Arnout Engelen wrote:
I'm often all too comfortable with odd experimentations.
Never heard of it but will investigate, thanks.
On Tue, Oct 05, 2010 at 03:12:12PM +1100, cal wrote:
Not sure - possibly the main reason for the post was to seek help in resolving
fallacies in my thinking. With a jack period of 128, a midi event associated
with frame 129 won't actually get applied to audio generation until the
generation of frames 257-384. On the other hand an event against frame 128 will
make it into the generation of frames 128-256. I figure that variability of
effective midi latency according to where the event sits in the period probably
qualifies asjitter?
[ ... ]
[ ... ]
latency in a soft synth (possibly yoshimi) ...
Latency? Or jitter? What seemed most promising was to break the audio
generation into smaller
blocks, applying pending midi events between blocks.
Isn't this what JACK does by choosing a period size? Splitting up the
processing withing a period like this seems odd. I'm assuming you're talking about ALSA MIDI,
here not JACK MIDI - is that
correct?
No, same applies to both.
Sadly, that
drags the creation and destruction of note objects into the
realtime jack process callback path. (...) Even with a pool of pre-allocated
note objects, it seems the amount of initialization code per note is still a
real limiting factor on how busy things can get before it all falls apart.
Of course allocation is forbidden in the process() callback, so that's off
limits anyway. Using pre-allocated notes, it sounds kind of unlikely to me
that initializing your note objects is the bottleneck. It seems more likely
to me that your way of getting the notes from the MIDI device (again - is this
ALSA midi? Can you share code/patches?) is not RT-safe. Have you tried running
it with http://repo.or.cz/w/jack_interposer.git ? To 'solve' ALSA MIDI jitter for JACK,
you'll have to receive ALSA MIDI messages
in a seperate thread, and make sure you apply them at the right position in the
process buffer based on a timestamp telling you when exactly the note event was
recorded.
A dedicated thread for alsa midi is what I've been using along. At the moment I'm
exploring using an alsa timer callback, and I quite like that.
With JACK MIDI, JACK takes care of the recording MIDI
events and timestamping
them in a sample-accurate way - all you'll have to do is make sure you do take
into account the timestamp to correctly place them in the process buffer.
I wish that that was "all you have to do". I think the crux of the biscuit is
that
given the way zyn/yoshimi does its magic, you simply can't have a midi event
changing note or controller parameters during the generation of a given period of
audio. Hence it's impossible to accurately honor the frame/time stamp of a midi
event. That's what drove drove the experimentation with splitting the audio
generation down to tighter blocks.
This will only 'solve' jitter though: a
latency proportional to the period size
is inherent to the way JACK works I'm afraid. Hence the desirability of tuning
your system to allow for low period sizes.
Precisely. I think the bottom line is that I've tried real hard, but the best you
can expect from zyn/yoshimi is determined by how tight you can get your jack period
with reliability - short of a radical re-engineering of the zyn internals, in which
case it'll be a whole new synth :-).
cheers.
1:58 p.m.
cal wrote:
On 05/10/10 18:51, Arnout Engelen wrote:
You seem to be assuming that when you are generating the sound for a
period, and when you find a new event, you have to put the event's
effect into the entire period, i.e., the event's note then starts at the
start of the period.
What you have to do is to remember the timestamp of the event (measured
in frames, like above), and then apply the event at that time in the
period that corresponds to the original time, relative to the period in
which it was received. From your example above, the audio data for the
event received for frame 128 should start at the last frame of the
period, i.e., frame 256, while the audio data for the event received for
frame 129 starts at the first frame of the next period, frame 257.
Jitter is defined as varying latency. You can remove it by taking the
worst-case latency (if it exists) (one period in this case) and applying
it to all events.
Regards,
Clemens
Latency? Or jitter?
Not sure - possibly the main reason for the post was to seek help in resolving
fallacies in my thinking. With a jack period of 128, a midi event associated
with frame 129 won't actually get applied to audio generation until the
generation of frames 257-384. On the other hand an event against frame 128 will
make it into the generation of frames 128-256.
3:09 p.m.
On 05/10/10 22:58, Clemens Ladisch wrote:
cal wrote:
In a perverse way, that's helpful - for me it confirms that I need to gain a
deeper & stronger understanding of zyn's current note on/note off processing
before I'm likely to achieve improvement.
On 05/10/10 18:51, Arnout Engelen wrote:
You seem to be assuming that when you are generating the sound for a
period, and when you find a new event, you have to put the event's
effect into the entire period, i.e., the event's note then starts at the
start of the period. Latency? Or jitter?
Not sure - possibly the main reason for the post was to seek help in resolving
fallacies in my thinking. With a jack period of 128, a midi event associated
with frame 129 won't actually get applied to audio generation until the
generation of frames 257-384. On the other hand an event against frame 128 will
make it into the generation of frames 128-256. What you have to do is to remember the timestamp of
the event (measured
in frames, like above), and then apply the event at that time in the
period that corresponds to the original time, relative to the period in
which it was received. From your example above, the audio data for the
event received for frame 128 should start at the last frame of the
period, i.e., frame 256, while the audio data for the event received for
frame 129 starts at the first frame of the next period, frame 257.
Jitter is defined as varying latency. You can remove it by taking the
worst-case latency (if it exists) (one period in this case) and applying
it to all events.
All genuinely helpful, thank you!
cheers, Cal
2:39 p.m.
On Tue, Oct 05, 2010 at 08:27:02PM +1100, cal wrote:
I like it, but I'm biased as i wrote it :)
Ah, then I misunderstood you there.
OK, now I understand what you meant better: splitting up the JACK period into
multiple subperiods is not something you like to do per se, but something that
makes re-using existing zyn/yoshimi code easier. That can make sense, yes.
On 05/10/10 18:51, Arnout Engelen wrote:
With 'make it into the generation' you mean the midi event associated with
frame 129 will end up in the beginning of frames 257-384 while an event
against frame 128 will end up in the beginning of frames 128-256? Then yes,
that 'variability of effective midi latency' is indeed exactly what jitter is.
On Tue, Oct 05, 2010 at 03:12:12PM +1100, cal
wrote:
Not sure - possibly the main reason for the post was to seek help in resolving
fallacies in my thinking. With a jack period of 128, a midi event associated
with frame 129 won't actually get applied to audio generation until the
generation of frames 257-384. On the other hand an event against frame 128 will
make it into the generation of frames 128-256. I figure that variability of
effective midi latency according to where the event sits in the period probably
qualifies as jitter? latency in a soft synth (possibly yoshimi) ...
Latency? Or jitter? is the
bottleneck. It seems more likely
to me that your way of getting the notes from the MIDI device (again - is this
ALSA midi? Can you share code/patches?) is not RT-safe. Have you tried running
it with http://repo.or.cz/w/jack_interposer.git ?
Never heard of it but will investigate, thanks. To
'solve' ALSA MIDI jitter for JACK, you'll have to receive ALSA MIDI messages
in a seperate thread, and make sure you apply them at the right position in the
process buffer based on a timestamp telling you when exactly the note event was
recorded.
A dedicated thread for alsa midi is what I've been using along. At the moment
I'm
exploring using an alsa timer callback, and I quite like that. With JACK
MIDI, JACK takes care of the recording MIDI events and timestamping
them in a sample-accurate way - all you'll have to do is make sure you do take
into account the timestamp to correctly place them in the process buffer.
I wish that that was "all you have to do". I think the crux of the biscuit is
that
given the way zyn/yoshimi does its magic, you simply can't have a midi event
changing note or controller parameters during the generation of a given period of
audio. Hence it's impossible to accurately honor the
frame/time stamp of a midi
event. That's what drove drove the experimentation with splitting the audio
generation down to tighter blocks.
Yes, that could be an interesting way to reduce (though not eliminate entirely)
jitter even at large jack period sizes.
Arnout
2:50 p.m.
On Tuesday 05 October 2010, at 14.39.25, Arnout Engelen <lad(a)bzzt.net> wrote:
[...]
Not only that. As long as the "fragment" initialization overhead can be kept
low, smaller fragments (within reasonable limits) can also improve throughput
as a result of smaller memory footprint.
Depending on the design, a synthesizer with a large number of voices playing
can have a rather large memory footprint (intermediate buffers etc), which can
be significantly reduced by doing the processing in smaller fragments.
Obviously, this depends a lot on the design and what hardware you're running
on, but you can be pretty certain that no modern CPU likes the occasional
short bursts of accesses scattered over a large memory area - especially not
when other application code keeps pushing your synth code and data out of the
cache between the audio callbacks.
--
//David Olofson - Developer, Artist, Open Source Advocate
.--- Games, examples, libraries, scripting, sound, music, graphics ---.
| http://olofson.net http://kobodeluxe.com http://audiality.org |
| http://eel.olofson.net http://zeespace.net http://reologica.se |
'---------------------------------------------------------------------'
Hence it's
impossible to accurately honor the frame/time stamp of a midi
event. That's what drove drove the experimentation with splitting the
audio generation down to tighter blocks.
Yes, that could be an interesting way to reduce (though not eliminate
entirely) jitter even at large jack period sizes. 
10 p.m.
On Tue, Oct 05, 2010 at 02:50:10PM +0200, David Olofson wrote:
Not only that. As long as the "fragment"
initialization overhead can be kept
low, smaller fragments (within reasonable limits) can also improve throughput
as a result of smaller memory footprint.
'Fragment initialisation' should be little more than
ensuring you have the right pointers into the in/out
buffers.
Depending on the design, a synthesizer with a large
number of voices playing
can have a rather large memory footprint (intermediate buffers etc), which can
be significantly reduced by doing the processing in smaller fragments.
Obviously, this depends a lot on the design and what
hardware you're running
on, but you can be pretty certain that no modern CPU likes the occasional
short bursts of accesses scattered over a large memory area - especially not
when other application code keeps pushing your synth code and data out of the
cache between the audio callbacks.
Very true. The 'bigger' the app (voices for a synth, channels for
a mixer or daw) the more this will impact the performance. Designing
the audio code for a fairly small basic period size will pay off.
As will some simple optimisations of buffer use.
There are other possible issues, such as using FFT operations.
Calling a large FFT every N frames may have little impact on
the average load, but it could have a big one on the worst case
in a period, and in the end that's what counts.
Zyn/Yoshimi uses FFTs for some of its algorithms IIRC. Getting
the note-on timing more accurate could help to distribute those
FFT calls more evenly over Jack periods, if the input is 'human'.
Big chords generated by a sequencer or algorithmically will still
start at the same period, maybe they should be 'dispersed'...
Ciao,
--
FA
There are three of them, and Alleline.
10:22 p.m.
On Tue, 5 Oct 2010 22:00:11 +0200
fons(a)kokkinizita.net wrote:
On Tue, Oct 05, 2010 at 02:50:10PM +0200, David
Olofson wrote:
I'm all in favour of a bit of dispersal.
When I started out with a Yamaha SY22 and Acorn Archimedes it was all
too easy to stuff too much down the pipe at once. However, doing some
experimenting, I was surprised at how much you could delay or advance
Note-On events undetectably although it depended to some extent on the
ADSR envelope.
I don't need to do that any more, but old habits die hard, so if I'm
copy-pasting tracks I tend to be deliberately a bit sloppy.
I'm also a bit puzzled by people complaining about jitter. I don't have
any exceptional kit, but in reality I can't say I've ever noticed it.
Latency yes, but that's easily corrected with a bit of post record
nudging.
--
Will J Godfrey
http://www.musically.me.uk
Say you have a poem and I have a tune.
Exchange them and we can both have a poem, a tune, and a song.
Not only that. As long as the
"fragment" initialization overhead can be kept
low, smaller fragments (within reasonable limits) can also improve throughput
as a result of smaller memory footprint.
'Fragment initialisation' should be little more than
ensuring you have the right pointers into the in/out
buffers.
Depending on the design, a synthesizer with a
large number of voices playing
can have a rather large memory footprint (intermediate buffers etc), which can
be significantly reduced by doing the processing in smaller fragments.
Obviously, this depends a lot on the design and
what hardware you're running
on, but you can be pretty certain that no modern CPU likes the occasional
short bursts of accesses scattered over a large memory area - especially not
when other application code keeps pushing your synth code and data out of the
cache between the audio callbacks.
Very true. The 'bigger' the app (voices for a synth, channels for
a mixer or daw) the more this will impact the performance. Designing
the audio code for a fairly small basic period size will pay off.
As will some simple optimisations of buffer use.
There are other possible issues, such as using FFT operations.
Calling a large FFT every N frames may have little impact on
the average load, but it could have a big one on the worst case
in a period, and in the end that's what counts.
Zyn/Yoshimi uses FFTs for some of its algorithms IIRC. Getting
the note-on timing more accurate could help to distribute those
FFT calls more evenly over Jack periods, if the input is 'human'.
Big chords generated by a sequencer or algorithmically will still
start at the same period, maybe they should be 'dispersed'...
Ciao,
11:55 a.m.
On Tue, Oct 05, 2010 at 09:51:35AM +0200, Arnout Engelen wrote:
Yes, but it provides a simple way to improve an existing
application using Jack-midi, withour requiring very deep
changes to the audio code to make it fully 'sample accurate'.
You will probably already have something like:
jack_process (nframes)
{
read_jack_midi_data(); // read full buffer, create new notes,
// mark those being released.
synthesize(nframes); // audio processing for current notes
}
The latter call will already accept smaller values of nframes,
so no changes at all should be necessary there.
So the only changes required would be to read_jack_midi_data(),
making it read the next required part of the midi buffer instead
of all of it, and then put a loop around the two calls, dividing
Jack's period into smaller ones.
An occasional non-realtime safe call will do less damage as when
using a real smaller period as its effect is amortised over the
full longer period.
This is how I added Jack-midi support to Aeolus. It has an internal
'period' of 64 frames which really can't be changed without rewriting
all of it. Reading the midi data in slices of 64 frames and alternating
this with the synthesis code provides a jitter of +/- 32 frames, which
is probably OK (at least it is for an organ).
Ciao,
--
FA
There are three of them, and Alleline.
What seemed
most promising was to break the audio generation into smaller
blocks, applying pending midi events between blocks.
Isn't this what JACK does by choosing a period size? Splitting up the
processing withing a period like this seems odd.
2:47 p.m.
On 05/10/10 20:55, fons(a)kokkinizita.net wrote:
On Tue, Oct 05, 2010 at 09:51:35AM +0200, Arnout
Engelen wrote:
Yes, but it provides a simple way to improve an existing
application using Jack-midi, withour requiring very deep
changes to the audio code to make it fully 'sample accurate'.
You will probably already have something like:
jack_process (nframes)
{
read_jack_midi_data(); // read full buffer, create new notes,
// mark those being released.
... improving the way it currently handles the create new + mark
released processing is what's needed, and from where I sit it's a
fairly daunting task. I'll probably have a go at it, ... but I'm not
a young man :-(.
What
seemed most promising was to break the audio generation into smaller
blocks, applying pending midi events between blocks.
[ But, but ... ] synthesize(nframes); // audio processing for
current notes
}
[ ... ]
This is how I added Jack-midi support to Aeolus. It
has an internal
'period' of 64 frames which really can't be changed without rewriting
all of it. Reading the midi data in slices of 64 frames and alternating
this with the synthesis code provides a jitter of +/- 32 frames, which
is probably OK (at least it is for an organ).
All up, serious food for thought and very much appreciated!
Thank you!
Cal
5155
days inactive
5155
days old
linux-audio-dev@lists.linuxaudio.org
10 comments
6 participants
participants (6)
-
Arnout Engelen -
cal -
Clemens Ladisch -
David Olofson -
Folderol -
fons＠kokkinizita.net