Re: [LAD] Non Session Management

On Tue, 2012-03-27 at 01:04 +0200, Emanuel Rumpf wrote: This exact problem came up when I set out to implement non-destructive plugin state saving in Ardour, with support for plugins referring to files (e.g. loaded samples). The obvious basic solution in a non-destructive context is to save each snapshot to its own directory. The tricky bits come when you have links in that directory to files. What you end up with is a few directories with specific roles. I'll try to document the path leading to this as tersely as possible. The plugin/host situation is analogous to app/SM, very closely with respect to what's on disk, not so much with respect to responsibilities and such. There are two kinds of file: * External files, e.g. something you loaded from the filesystem somewhere. There are assumed to be immutable. * Files created by the plugin, which may be different each save As I have said a few times on this list, for various reasons I think straightforward and transparent symlinks are the best way to refer to external files. I'll just take this as given/obvious and not bother rehashing the argument (erecting a bunch of protocol and/or file format junk only *adds* problems). So, you want to save state, and the plugin refers to a bunch of files. As mentioned above, this can mean a lot of references to one file. Since these could be massive, redundant copies must never happen at all. For ease of moving things, or resolving them if they break, we want a minimal number of actual links to an external file, i.e. 1 (this is also necessary for archival, see below). So, we don't want every single state snapshot directory to have a link to /media/bigfile.wav. To avoid this, make a directory specifically for all links to external file, and link to those links instead. This localizes all links to files outside the session. The other case is files written to during execution, e.g. recorded waveforms. Here, on each save, you need to check if the file is actually any different, and make an actual copy if so (so the original can continue to be used) in the state snapshot directory. During run time, all the created files live in a "scratch" directory, which you can preserve, or just throw out when you close since copies have been made for every snapshot. Working with the requirement that the plugin's file namespace must not be messed with (e.g. if you save foo.wav to your state directory, it should actually be called foo.wav), you end up with: 1) The "file directory". This is where the plugin creates files, whenever. It can be considered run-time scratch. 2) The "copy directory". This is where copies of things in the file directory are made, to preserve their state at a particular instant in time. This mirrors the structure of the file directory, but appends .2 or .3 etc. for the various snapshots of a file 3) The "save directory". This is the directory of a particular state snapshot. It can contain whatever. 4) The "link directory". This is where all links to external resources live. Any state snapshot that refers to an external file actually links to a link here. This is in the Lilv API. It might seem complicated, but it is only optionally this powerful, if you don't care about non-destructive saving or avoiding copies you can simply use one directory for everything. I think I can make a reasonable argument that this is the minimal number of directories that meet the desired requirements: A) The file directory and copy directory can not be the same directory, because the copies would pollute the plugin's file namespace and possibly clash. It would also make it difficult to know what files are actually needed by the session, since some may be scratch. The files directory is unique in that its contents may be safely deleted. B) The copy directory and save directory can not be the same directory, since multiple saves may refer to the same file with the same state. Without the copy directory this would require two copies of the same file. C) The link directory must be separate to avoid having a large number of links to the same external file. This is convenient in general (only one link to potentially fix), but required for archival. An archival tool (e.g. tar -h) would actually copy the file to the links location. Without the link directory this would produce an archived session with many copies of the same (possibly massive) file. D) The save directory is inherently where one save takes place, its existence is a given. Figuring this all out was a long process of trial and error, but now that I write it down it seems clear it can't be simplified without failing to meet a requirement. I am all ears for arguments to the contrary, though. This scheme would need some slight tinkering to tolerate external files that change, but this would require keeping copies of system files around just in case, not just when you archive. That is, ANY external files used EVER would have to be copied in case they change. I don't think this is acceptable, and plugins have no business modifying external files anyway. Apps that want to get fancy and share files may want a solution to this, but particularly since shared mutable files is almost certainly crazy anyway, one app should simply "own" any such files. It may tell other hosts about them if it pleases, possibly via OSC. I don't think the session manager itself should be bloated out with a bunch of unnecessarily shared file database stuff to support a rather esoteric ability that can easily be achieved without doing so. In other words, if apps want to talk to each other and share file paths, that is their business. -dr