On Wed, Nov 2, 2011 at 10:25 AM, Paul Davis <paul(a)linuxaudiosystems.com>wrote;wrote: thanks Paul, I think I was unclear. If I understand you correctly, then I meant transforming high level data. I'm making a CV style step sequencer for live looping, so I'm talking about having Python be used to do things like apply transformative routines to material that is in the sequences but is not yet in the audio output chain. Stuff like, when I hit this midi key, run this routine over the sequence data. These Python transformations are meant to be run at lower priority, ie if the engine wants the processor to spit out the next sample while the transformation is part way through, that's fine, it gets interrupted. I wasn't intending to use Python to apply dsp to signals going out in realtime, that would likely be accomplished using either the STK or embedded Csound instances. Does that sound more feasible? BTW, excuse my ignorance, but what is RCU? I found some blog posts on Ross Bencina's (sp?) site about making sure communication between the high priority engine and lower priority processes works right, but I'm hoping to find more concrete examples of this, and figure out how to do it between python and c++. thanks for your help, Iain

A good starting point for reading is on Software Transactional Memory (STM): http://en.wikipedia.org/wiki/Software_transactional_memory Jeff On 2011-11-02, at 10:40 AM, Paul Davis wrote:

Thanks Paul. I'm wondering if it might be easier to begin with to have my python side only read the shared data and send messages to the c engine for writes. Is there a relatively straightforward way to have a python process have read access to a block of memory that the c engine has full access to? Would this simplify the concurrency issues? thanks for the tips everyone. Iain

On 11/02/2011 07:26 PM, Iain Duncan wrote: The only example I know of is zynjacku: http://home.gna.org/zynjacku/ Implementation is C instead of C++... but same principles. -gabriel

On Wed, Nov 2, 2011 at 8:26 PM, Iain Duncan <iainduncanlists(a)gmail.com> wrote: I have some code somewhere that uses Boost.Python to expose a Jack MIDI port. So I can generate MIDI in Python, then pass it to the jack thread vi jack_ringbuffer, then out the app's MIDI port. Using pygame, for example, I wrote a simple python app that turned joystick button presses into MIDI notes so I could play drums with my gamepad. I also interfaced it to the mingus python package [http://code.google.com/p/mingus/] so that I could write mingus apps that send midi data to the jack midi app of my choice instead of mingus' built-in fluidsynth. I have a simple llittle pygtk app that draws a button for each chord in the chosen key and sends that chord out as midi on button presses. Fun! I hope to one day clean it all up and put it somewhere, plus add support for passing NumPy arrays to the jack thread to send out an audio port, yada yada, more hot air and empty promises, etc... There was another thread a few months back about message passing in C++ for jack apps. It might be worthwhile to revisit that thread in the context of wrapping things with Boost.Python.

Thanks everyone. Yeah, I did think of using osc to make some kind of message passing protocal, but also wondered if that would get restrictive. Wow, this thread gives me a lot to think about. Paul, would it be practical/possible to have the python process allowed to read the shared memory only? Does that simplify the locking problems, or not really? I could imagine that I could do what I want pretty well by doing read-copy-send messages back to engine. But obviously some of you have thought a lot harder about this. thanks iain On Thu, Nov 3, 2011 at 2:47 PM, Harry van Haaren <harryhaaren(a)gmail.com>wrote;wrote:

On Thu, Nov 3, 2011 at 9:22 AM, Harry van Haaren <harryhaaren(a)gmail.com> wrote: no, using a protocol like OSC doesn't solve the basic problem: making changes to data structures used by the RT code when those changes can't be done with RT constraints (e.g. memory allocation). you still need some code that runs outside RT constraints to modify the data before giving the new version to the RT code to use. --p

On Thu, Nov 3, 2011 at 1:39 PM, Paul Davis <paul(a)linuxaudiosystems.com>wrote;wrote: Yes, sorry must not have made myself clear :) Currently using the jack ringbuffer as a RT queue. Thought of a nice abstraction of the "one way only" problem too: Just wrap two ringbuffers into a single class, and then have two set / get functions. Basically one "RtQueue" instance can pass messages back and forth between two threads lock free. I'd like to improve it so that one get / set method is there, and its thread-aware so it will automatically push / pull to the right ringbuffer. But that's a touch hard for me as I'm using GLib threads and finding which thread is running is something i can't do yet :D -Harry

On Thu, Nov 3, 2011 at 8:25 PM, Paul Davis <paul(a)linuxaudiosystems.com>wrote;wrote: Good point, and in some places I do find myself iterating lists into a vector in a non-RT context, and then sending the vector as a blob via OSC. It is a drawback of this separation.

On , Iain Duncan <iainduncanlists(a)gmail.com> wrote: When sending a "blob" of binary OSC data, it is required that this data is sequential in memory, as in an array. Linked lists don't nessiarily have this property, and hence you can't blob send them. So the nasty fix is to iter over the list, and put each event into a vector, and then blob send the memory starting at vectorName[0] until vectorName[0] + vectorName.size().

2011/11/3 Paul Davis <paul(a)linuxaudiosystems.com>om>: While this is restrictive, in the way you mention, I think it's a welcome simplification ( compared to implementing a real-time-capable-linked-list + other rt-structures ), if your use-case doesn't require direct calls to the list (for any reason), then you could request insert/remove/update operations through the rt-ring-buffer. -- E.R.

On Fri, Nov 4, 2011 at 10:23 AM, Paul Davis <paul(a)linuxaudiosystems.com>wrote;wrote: Sorry, what do you need instead? ( trying hard to absorb all this.. ) thanks iain

either Sounds like it will be worth learning about both. I gather the Ardour source is a good example of RCU correct? Can anyone point me at documentation or examples of a lock free data structure? thanks Iain

2011/11/4 Paul Davis <paul(a)linuxaudiosystems.com>om>: Right. What I meant is: to enqueue the required operations in a realtime-queue (ringbuffer), then read that queue from a non-rt thread in order to update the non-rt list. This implies OTOH, that list-actions take place with a delay (at least not instantly), but may work in some situations, e.g. recording an audio stream. @ Iain and others Before you start to implement your own, which can take as much time as your whole project ... Here is a list of some libs I found, that implement lock-free structures, STM, etc. http://wiki.linuxaudio.org/wiki/programming_libraries#lockfree_non-blocking… If anything is missing, let me know, or edit yourself. -- E.R.

this approach requires that all your data structures can be Just wondering if I can get a clarification, this is the internal audio/sequence data for the engine that needs to be representable as POD? The reason for this is so that when inserts happen the RT thread is not dealing with memory management for a linked list? Am I correct in understanding that it's the memory allocation part of creating new pointers that needs to be avoided in the RT thread in case it blocks or gets stalled by the OS? Does this mean that if I can get away with some kind of data scheme for the engine that is not dependent on the pointer management that it will be a lot easier to update based on incoming queued messages? thanks Iain