[LAD] [A plugin scanner:] Cache text file formats: rdf, ttl, custom xml?

[Robin Gareus]

On 10/29/2018 06:52 AM, Tim wrote:

> On 10/29/2018 12:40 AM, Hermann Meyer wrote:
>> Downside of cached information is, that it could clash on plugin load
>> when the plugin have changed it's ports (updated).
If that happens you have a much bigger issue than stale caches.

Released plugins should not change ports or behavior in a way that is
not backwards compatible. Doing so would break existing sessions using
the plugin.


>> So when you save plugin information on your own, you need to check
>> before you at least load the plugin, if the cached information is
>> still valid.
>
> 
> Yes that was of great concern from the start of this effort.
> 
> Scenario 1: While the the program is not running, the user
>  installs, moves, removes, or upgrades a plugin, then runs
>  the program.
> I have an idea at program startup to compare the date/time
>  of each given plugin path with the one of the cache file.
> If different, reload the cache.

I suggest to prefer a checksum or hash (at least additionally), or use
inotify or some existing watch mechanism.

Ardour uses stat()'s mtime. There are various issues with this:
e.g. zip files do not include timezone, some tar archives include
strange timestamps, installers on other platforms do strange things, and
some users toy with clocks on occasion.
Edge cases like these have lead to countless headaches in the past.


> Scenario 2: While the the application is running - perhaps already
>  for hours into a project - the user decides to install, move, remove,
>  or upgrade a plugin. Watch out !
> I have an idea to 'watch' all the given plugin paths for changes
>  and take appropriate action: Crash ;-)
> I don't think the scenario is too drastic though.
> The plugins already loaded into memory should still hopefully be OK.

yep.

On Unix systems already loaded .so will be kept in memory. On Windows
you cannot write/replace to a file that is currently opened.

You can skip and postpone scanning of plugins that are currently in use
until the next session load.

> It might not be wise to alter my data or info of an already loaded
>  plugin if it is deleted or removed or upgraded, but at the very
>  very least I would like to be able to support loading a
>  'brand new' plugin if it suddenly appears.

Why? I've never seen someone installing/removing software while
recording or mixing. It also sounds like a bad idea to me.

What happens if you run two instances of the same DAW or two different
programs that share the plugin cache?

Do all programs watch for changes and use lock-files to prevent
concurrent scanning?

One solution is to ask a user to directly or indirectly initiate a scan.
Directly by pressing a button or indirectly by opening a session. These
actions are rarely performed concurrently.


> 
> Also IIUC sometimes plugin ports can be created on-the-fly? Ouch.
> 

Do you need to know the ports a-priori?

The host needs to load the plugin anyway to insert it. So the only
interesting part to cache is

 * Plugin-Name
 * Plugin-Author
 * Factory & User Presets
 * Category/Tags (if any)
 * Description (if any)

I/O and Control Ports are not usually something that's worth caching,
except perhaps for "has MIDI I/O" (to detect Instruments for plugins
that don't specify a category).

What other information are you interested to cache? Is there a benefit
to caching port names, default values, for each plugin?


> About my cache format: Since the highest denominator (most advanced)
>  in all these plugin types would be LV2, I think it would be really
>  cool if the Turtle tools or libraries could report ALL types
>  of plugins and generate new ttl files on the fly from found plugins.

RDF is generic, it can describe pretty much anything. I thought there is
already a LADPSA ontology. Also LADSPA uses RDF for presets.

TTL is just a RDF serialization. There is also some discussion to allow
JSON-LD as alternative to turtle for LV2.


LV2's ontology provides for a nice description of the basics, it really
uses  foaf[1] and doap[2,3] for the basics.  This leaves some
classification (categories, tags) and  a presets-name to ID map.

I have no opinion about the de/serialization format. turtle is fine with
me, as would be JSON-LD. Both can be efficiently parsed (low CPU, low
memory, serial I/O) and satisfy requirements (semantic description,
ordered data structure).

> Then we could all simply use our same LV2 discovery codes to examine
>  all plugin types found.

I'm not sure if that's very likely, nor practical. While serd/sord are
generic to deal with turtle/RDF, liblilv is rather specific for LV2. The
mechanism to scan/discover plugins and presets is rather different for
each standard.

Realistically, these days the only interesting plugin formats are LV2,
VST and AudioUnit.

VST does not even specify a system-wide install location for discovery.
The AU format is abstracted by Apple, you can't directly interact with
it but have to use Apple's libs. This leaves LV2.


On a related note: Why stop at scanning? There could be a standalone
host with a protocol that interacts with the DAW (at the expense of a
context switch): Much like Bitwig does for sandboxing plugins.

It could be a stripped down jack-application with an additional
state/control API.


2c,
robin

[1] http://xmlns.com/foaf/spec/
[2] https://en.wikipedia.org/wiki/DOAP
[3] http://usefulinc.com/ns/doap#