[LAD] Portable user interfaces for LV2 plugins

Stefano D'Angelo zanga.mail at gmail.com
Tue Mar 1 13:13:10 UTC 2011

2011/3/1 David Robillard <d at drobilla.net>:
> On Mon, 2011-02-28 at 21:51 +0100, Olivier Guilyardi wrote:
>> (split from: RDF libraries, was Re: [ANN] IR: LV2 Convolution Reverb)
>> On 02/26/2011 11:35 PM, David Robillard wrote:
>> > At this very instant, on a particular device, browser might not be up to
>> > snuff.
>> >
>> > Personally I'm more interested in better long-term investments, and the
>> > browser is only going to get better - and it's getting better very, very
>> > quickly. This stuff is moving way too fast to throw out all the wins and
>> > ease of browser UIs, IMO.
>> > The ultra-portability is a really lucrative feature. Being able to
>> > control plugins over the network from anything with a web browser
>> > without requiring any UI-client side specific code whatsoever is a whole
>> > lot of awesome. Even for desktop software - have a nice phone or tablet?
>> > Go to http://yourmachine:12345 (or whatever) and there's the UI. No
>> > screwing around with phone/tablet software whatsoever.
>> I tend to believe that, generally, what is supposed to work in all cases doesn't
>> work well in specific cases. It may do the job somehow, but it's not really
>> adapted. And if this results in poor acceptance, then your investment is lost.
>> Portability is good, but if you go too far with it, you lose a lot of API and/or
>> platform specific features and optimizations.
> </hand-waving>
>> Also, I don't see what's so easy with browsers. I've done web development for
>> years, and compatibility problems are the rule.
> Never said it was easy, but requiring a modern browser is still much,
> much more portable and less annoying than requiring a bunch of specific
> native code on every device. This is not a "web site" that needs to work
> in IE6 or whatever.

If we exclude older IE releases it all gets a lot better :-)

>> > Just want the UI on the same machine? Do the same in your browser.
>> I don't see why this is so crucial for plugins.
> This stuff is more appropriate for controllers. Faders, knobs, buttons,
> grids, loop sequencers, etc. Maybe you personally don't care to control
> audio software with a tablet (or any machine on which you can't install
> a bunch of native LAD crap) but there's no question that a lot of people
> do.
> Personally I don't care about high performance native UIs that draw
> waveforms or whatever, because the last thing I want to be doing (or
> anybody wants to watch) is clicking around on a damned screen when I'm
> trying to play. 99% of that stuff is worthless fancy bling intended for
> back of the box screenshots, if you ask me. Plain old lines and flat
> rectangles are not only as good - but better - for tactile UIs actually
> designed for playability/readability.

True for live usage, not really for offline processing, recording
sessions, etc. At least not in the long run. Think spatialization,
EQs, physics-related stuff, scientific usage and, somewhere in the
future, sound analysis (spectrograms, etc.), highly interactive stuff.

WebGL FTW???

>> It's true that browsers are evolving fast, but right now you can't even get a
>> VU-meter to update fast enough in a portable manner. Browsers are not adapted to
>> such things as live frequency curves and other powerful audio UIs.
> Obviously, for the sort of thing where you need a rapid update VU meter,
> waveforms, etc, using web stuff is not ideal (though see my earlier
> argument about short-sightedness re: the rate of progress of browser
> tech).
> VU meters from an instrument perspective don't need to be fast, you just
> send peaks. A vague sense of levels that (crucially) shows all clips is
> totally doable remotely in a browser (but that said, this again isn't
> really what I'm shooting for).

I actually am quite impressed with YUI, it seems very fast in doing
non trivial stuff. However, I couldn't imagine a waveform editor, to
say, in browsers yet.

>> > I understand your priorities might be slightly different, since you're
>> > trying to push Android software in the market, but that's my position.
>> >>From a makin' money makin' apps perspective, a free iPhone/iPad port
>> > sure doesn't hurt, though...
>> I don't think it's a matter of priorities here. We have different opinions on
>> what plugin UIs could be on mobile devices.
>> Actually, on current mobile platforms, when one wants a portable UI, there is an
>> alternative to Web UIs: OpenGL. This runs everywhere, and as smoothly as can be.
>> All you need is the plugin to expose draw() and mouse motion functions.
> Yep... and it's not remote, and involves writing a bunch of platform
> specific native code. The whole point is avoiding that.
> Do I think GL is the thing to use _if you want to, and can deploy,
> implement device specific native code for the UI_? Absolutely. Is that
> suitable for all (or even most) cases where browser UIs shine? Nope.
> As one example, I want to have a machine controlling the audio rig, have
> people arrive with their tablet (or whatever), go to a particular
> address and participate in the jam. This is be a pretty
> awesome/novel/unique possibility. Non-realtime audio is even a
> possibility if their device can do such things. Obviously, the only way
> of doing this is web UI. As a nice plus, when you do it that way, hey,
> you get a PC appropriate network transparent UI for free. From the
> perspective of someone who needs this anyway, some very tangible reasons
> would be needed to make rewriting the whole UI in GL as well not an epic
> waste of time. Note that most of realising this dream will be done by
> the host, only certain plugins would need special web UI fragments. The
> rest just need to provide sufficient information for the host to make
> sense of their parameters (as they need to regardless).

/me thinks, at this point, about a possible WebUI decorator plugin
with interactive design possibilities...

(then my mind pushes it too far: let's integrate with Firebug or something. :-P)

> If you want to do some sort of experimental fancy 3D plugin UIs rendered
> in the same 3D universe or whatever (right now, i.e. not using webGL),
> where it is necessary for a plugin to have special UI code (i.e. the
> host can't generate it) sure, this is not the way to go. Use
> GL/Clutter/whatever. Unless you actually need the performance advantage
> of native GL, though, browser is better.

Uhm, GL is actually embeddable in most semi-serious toolkits (even SDL
should allow that IIRC). Should we go for a GL/WebUI approach?

>> Using such functions glScissor(), it would even be possible to embed a plugin UI
>> in a host UI while making sure that it doesn't draw out of its bounds.
>> A toolkit on top of OpenGL may often be useful especially for drawing texts and
>> widgets. For example, there is Clutter: http://www.clutter-project.org/
>> But this is no constraint, every plugin would be free to use the toolkit of its
>> choice internally when rendering within its draw() function.
>> There are subtleties though. For example on Android, there are so-called screen
>> densities, theoretically ranging from 120dpi to 320dpi according to the device.
>> But that's easily solved with OpenGL, the host can scale the displayed plugins
>> if needed. The plugins do not need to know what the ratio between a point and a
>> pixel is.
>> And if there existed some specialized toolkit for audio UI development, and even
>> a UI builder in the lines of JUCE's Jucer, then creating portable and
>> high-performance plugin UIs would become very easy.
>> That said, I have also investigated other solutions for simple portable plugin
>> UIs, but I think that OpenGL is worth considering.
> I fully support people doing openGL (or clutter, or whatever) UIs,
> nothing's stopping you. It won't solve the problem I am trying to solve,
> though. This is another example of why LV2 doesn't cram a toolkit down
> your throat. It is unclear right now whether the web UI stuff will even
> have anything to do with the UI extension.

/me: GUI decorator plugin #2 - give it a bunch of metadata (and maybe
fonts image), it will create the GUI on the fly for you.

> Anyway, the more interesting/important/pressing issue is how the UI
> communicates with the plugin, because that part really should be the
> same in both cases, and a solution is needed for currently existing
> things that are kludging around the lack of it with e.g.
> instance-access. Solution here coming soon.



More information about the Linux-audio-dev mailing list