Yes, that's certainly feasible. RCU = Read-Copy-Update is a software
pattern for dealing with situations where you need to update a complex
data structure while it is in use. The general approach is to make a
copy, modify the copy, and the atomically (normally) swap a pointer to
the original for a pointer to the new one. somehow you have to clean
up the old one.

you won't find this written up in any texts on programming, i think.
its in wide use in the linux kernel, and there have been attempts to
create some general purpose user space libraries that do it too. my
own sense is that almost every implementation of RCU will end up being
incredibly context (app) dependent. we use it a lot in ardour. its
quite complex.

Thanks Paul. I'm wondering if it might be easier to begin with to have my python side only read the shared data and send messages to the c engine for writes. Is there a relatively straightforward way to have a python process have read access to a block of memory that the c engine has full access to? Would this simplify the concurrency issues?

thanks for the tips everyone.
Iain