[LAD] [A plugin scanner:] Cache text file formats: rdf, ttl, custom xml?

Tim termtech at rogers.com
Mon Oct 29 18:07:34 CET 2018

On 10/29/2018 10:34 AM, Robin Gareus wrote:
> On 10/29/2018 06:52 AM, Tim wrote:
>> On 10/29/2018 12:40 AM, Hermann Meyer wrote:
>>> Downside of cached information is, that it could clash on plugin load
>>> when the plugin have changed it's ports (updated).
> If that happens you have a much bigger issue than stale caches.
> Released plugins should not change ports or behavior in a way that is
> not backwards compatible. Doing so would break existing sessions using
> the plugin.
>>> So when you save plugin information on your own, you need to check
>>> before you at least load the plugin, if the cached information is
>>> still valid.
>> Yes that was of great concern from the start of this effort.
>> Scenario 1: While the the program is not running, the user
>>   installs, moves, removes, or upgrades a plugin, then runs
>>   the program.
>> I have an idea at program startup to compare the date/time
>>   of each given plugin path with the one of the cache file.
>> If different, reload the cache.
> I suggest to prefer a checksum or hash (at least additionally), or use
> inotify or some existing watch mechanism.
> Ardour uses stat()'s mtime. There are various issues with this:
> e.g. zip files do not include timezone, some tar archives include
> strange timestamps, installers on other platforms do strange things, and
> some users toy with clocks on occasion.
> Edge cases like these have lead to countless headaches in the past.
>> Scenario 2: While the the application is running - perhaps already
>>   for hours into a project - the user decides to install, move, remove,
>>   or upgrade a plugin. Watch out !
>> I have an idea to 'watch' all the given plugin paths for changes
>>   and take appropriate action: Crash ;-)
>> I don't think the scenario is too drastic though.
>> The plugins already loaded into memory should still hopefully be OK.
> yep.
> On Unix systems already loaded .so will be kept in memory. On Windows
> you cannot write/replace to a file that is currently opened.
> You can skip and postpone scanning of plugins that are currently in use
> until the next session load.

Yes, that's what I was thinking.

>> It might not be wise to alter my data or info of an already loaded
>>   plugin if it is deleted or removed or upgraded, but at the very
>>   very least I would like to be able to support loading a
>>   'brand new' plugin if it suddenly appears.
> Why? I've never seen someone installing/removing software while
> recording or mixing. It also sounds like a bad idea to me.

There's the inconvenience of having to shut down and
  restart the program. I've had a couple of requests for that.
Would be nice since sometimes users just want to experiment
  with all the various plugins installing and trying each one out.

> What happens if you run two instances of the same DAW or two different
> programs that share the plugin cache?
> Do all programs watch for changes and use lock-files to prevent
> concurrent scanning?

Hm yeah good point. Should be do-able though.

> One solution is to ask a user to directly or indirectly initiate a scan.
> Directly by pressing a button or indirectly by opening a session. These
> actions are rarely performed concurrently.
>> Also IIUC sometimes plugin ports can be created on-the-fly? Ouch.
> Do you need to know the ports a-priori?
> The host needs to load the plugin anyway to insert it. So the only
> interesting part to cache is
>   * Plugin-Name
>   * Plugin-Author
>   * Factory & User Presets
>   * Category/Tags (if any)
>   * Description (if any)
> I/O and Control Ports are not usually something that's worth caching,
> except perhaps for "has MIDI I/O" (to detect Instruments for plugins
> that don't specify a category).
> What other information are you interested to cache? Is there a benefit
> to caching port names, default values, for each plugin?

Absolutely right. I went a bit overboard with this, storing much
  more information than I really needed. Sort of to the point of
  obsessing over it. And to what end? Once instantiated, we re-gather
  all that information from the plugin anyway. So a lot of it is useless.
I should probably stop where I am and just finish up what I've got.

>> About my cache format: Since the highest denominator (most advanced)
>>   in all these plugin types would be LV2, I think it would be really
>>   cool if the Turtle tools or libraries could report ALL types
>>   of plugins and generate new ttl files on the fly from found plugins.
> RDF is generic, it can describe pretty much anything. I thought there is
> already a LADPSA ontology. Also LADSPA uses RDF for presets.
> TTL is just a RDF serialization. There is also some discussion to allow
> JSON-LD as alternative to turtle for LV2.
> LV2's ontology provides for a nice description of the basics, it really
> uses  foaf[1] and doap[2,3] for the basics.  This leaves some
> classification (categories, tags) and  a presets-name to ID map.
> I have no opinion about the de/serialization format. turtle is fine with
> me, as would be JSON-LD. Both can be efficiently parsed (low CPU, low
> memory, serial I/O) and satisfy requirements (semantic description,
> ordered data structure).
>> Then we could all simply use our same LV2 discovery codes to examine
>>   all plugin types found.
> I'm not sure if that's very likely, nor practical. While serd/sord are
> generic to deal with turtle/RDF, liblilv is rather specific for LV2. The
> mechanism to scan/discover plugins and presets is rather different for
> each standard.
> Realistically, these days the only interesting plugin formats are LV2,
> VST and AudioUnit.
> VST does not even specify a system-wide install location for discovery.
> The AU format is abstracted by Apple, you can't directly interact with
> it but have to use Apple's libs. This leaves LV2.
> On a related note: Why stop at scanning? There could be a standalone
> host with a protocol that interacts with the DAW (at the expense of a
> context switch): Much like Bitwig does for sandboxing plugins.
> It could be a stripped down jack-application with an additional
> state/control API.

Absolutely. I mentioned this in our forum. This thing could grow
  from just a scanner to an actual run-time sandbox, since I now have
  the framework for running in an external process.

Of top concern is how to share data such as audio in/out buffers.
For inspiration I might look at the old FST or DSSI-VST or
  even Jack itself about how to share data between processes.
MusE has some very fine MPSC ring-buffer and multi-threaded
  memory C++ templates and so on. Hoping to leverage them.

Thanks for the help!

> 2c,
> robin
> [1] http://xmlns.com/foaf/spec/
> [2] https://en.wikipedia.org/wiki/DOAP
> [3] http://usefulinc.com/ns/doap#
> _______________________________________________
> Linux-audio-dev mailing list
> Linux-audio-dev at lists.linuxaudio.org
> https://lists.linuxaudio.org/listinfo/linux-audio-dev

More information about the Linux-audio-dev mailing list