[LAD] Portable user interfaces for LV2 plugins

Olivier Guilyardi list at samalyse.com
Wed Mar 2 13:20:30 UTC 2011

On 03/02/2011 12:22 AM, David Robillard wrote:

> On Tue, 2011-03-01 at 19:36 +0100, Olivier Guilyardi wrote:

>> Hmm, what platform specific native code? In my idea, the host would handle
>> setting up the GL viewport and such.
> ... "the host would <platform specific native code>" ;)

No, not really.

Apart from mobile web apps, a way to build cross-platform apps is using a
lightweight toolkit, with OpenGL for all graphics, and a couple of other entry
points such as an audio I/O callback, etc..

With this method, notably used by game devs, there's one code base, with thin
platform drivers.

So, with such portable hosts, the platform specific native code goes into the
toolkit. And both the host and plugin would be rather platform-agnostic.

> Going to something in a web browser doesn't involve any of that. It
> doesn't involve anything on the "thing with a UI on it" whatsoever
> except "go to this URL in your web browser". i.e. no platform specific
> native code (including GL) is required, at all, to operate the UI. It'd
> work on a hypothetical(*) tablet you can't even install software on
> whatsoever.

Okay, I see. But we are talking about plugins, are we? To me a plugin is a small
module which embeds in a host UI or appears as a small window on top of it.

What you're talking about makes sense if the host itself runs in the browser...
But switching from a native UI to a browser repeatedly is going to look rather
odd IMO. And if your idea is mainly intended for browser-based hosts then it
will not integrate properly with a large range of existing software.

That said, for instance on Android, you can embed a small WebView into the UI of
a host. So it could make sense. But (http) network operations are going to bring
some more overhead. All of this really isn't going to be smooth.

In order to avoid network operations, on Android, you could expose some of your
Java classes as Javascript objects into the WebView, this is entirely possible.
But now that's going to be a lot of native code for binding say a slider to
javascript and then to java and then down to the native plugin...

>>  The plugin would expose a draw() callback
>> in which it would only perform standard OpenGL calls.
>> I'm not an expert about GL portability, but AFAIK this might allow the UI
>> provided by a plugin to run unchanged on Linux, iOS, Android, Mac, Windows, etc..
> Probably, since the host deals with all the non-portable things. What
> you describe here sounds like how a GL UI type should indeed work. It
> would be better if there was some equivalent of a "widget" the UI could
> return, but I guess that is not the case for GL.

The UI would draw into a rectangular area which could:
- either be embedded in a larger GL scene, if the host UI is GL based,
- or be displayed by QGLWidget, rendered into a GdkGLWindow, etc..

Also, what's rather funny is that there would be some kind of symmetry between
the run() and draw() functions exposed by a plugin. One for audio, the other for
graphics/video... And both called in a cyclic manner by the host.

> So far, nobody has showed up with a desire to write a GL plugin UI.
> Extensions don't get invented until there's an actual use for them. If
> someone does, drop by lv2-devel, and we'll get this done (or don't, and
> just do it).

Yes, I've developed a long time for the desktop, and I never used OpenGL before
I got myself into mobile development. Now that I'm familiar with it, I quite
enjoy it, and am amazed by the perspectives in terms of portability.

> The next SLV2 will abstract away UI types, so if/when this gets
> invented, it will magically just work in existing hosts, since embedding
> GL in most toolkits is possible.

Sounds good. This is where LV2 extensibility shows some of its strengths I guess.

>> I'm confused now. You are talking about UIs automatically built by the host from
>> the plugin parameters, ports, metadata, etc..
> Yes, partially...
>>  To me, this is not a plugin UI,
> ... except a plugin could provide such a (plugin specific) UI as well.
> My "dirg" plugin provides a web-based grid UI in the browser, for
> example. Currently it just does this by running its own web server, but
> this needs to be pushed up to the host for obvious reasons.

Hmm, own web server, I don't know really.. This sounds a bit bloated to me.

>> this is the absence of plugin specific UI, and the kind of thing which you can
>> already do with LADSPA plugins, etc.. So, automatically generated web UIs from
>> plugin metadata is of course possible, but I wasn't talking about that.
>> I'm talking about those cases where the plugin needs to provide a specific UI,
>> as it's currently being done with LV2 using GTK, for example in Calf plugins,
>> etc.. But here you mention a mix of automatically generated UIs and "special web
>> UI fragments" provided by the plugin.
> "special web UI fragments", or, in other words, a plugin specific UI.
> Yep.

And how does one draws a frequency curve, even one which doesn't update in
realtime? With SVG? Do you see how heavy this is all going to be?

>> Regarding remote controllers.. Well this is already in the wild, tablets
>> included, with OSC and MIDI. For example, there is TouchOSC (sorry, not free,
>> but a great project) which even has a UI editor.
> Same kind of realm as this stuff, yes, except the UI design is
> integrated with the host and plugins, and is roughly infinity times
> slicker as a result.

Slicker for who? The whole point of controllers is that the user can choose the
one he/she likes. Hardware controller, tablet-based controller with knobs, or
sliders, or pad, or even one which uses the device sensors such as the
accelerator, the compass, etc..

>> I think that I understand you idea of doing this with a browser. No need to
>> setup OSC, ports, map URIs, etc.. But this sounds like a web-based controller
>> project. It's application level IMO, not sure it belongs to LV2, apart from the
>> UI fragments idea, but this all sounds like fundamental research to me.
> My LV2 host /is/ an LV2 plugin ;) Think about the implications of that
> for a minute and a big lightbulb should go off (Hint: Want to make
> plugins with universal UIs? Don't want to write code?)

Not sure what you mean here, but I'm a big No Silver Bullet fan, and generally I
don't believe in no-need-to-code-anymore claims.

> Is it about web-based controllers on a tablet or whatever? Yes. But it
> is not /only/ about that. If you have such a UI, well, it works just
> fine on the PC too. One UI written, both problems solved, objectively
> better solution - unless you need the performance of OpenGL. I don't
> deny that sometimes you do, but personally I'm more into making music
> and less into fancy screenshots and idiotic pixmaps of physical boxes or
> whatever.

Now, first, I'll repeat what I said earlier: "I tend to believe that, generally,
what is supposed to work in all cases doesn't work well in specific cases. It
may do the job somehow, but it's not really adapted. And if this results in poor
acceptance, then your investment is lost."

Also, I'm into music too. The greatest reward (apart from money ;) I currently
get when working on my product is enthusiastic feedback from musicians.

Now, think of a music instrument. Do you mean that a grey guitar with
unpersonalized design is what musicians need? I strongly disagree. And that's
why I think that powerful UIs with their own style can be a good thing. It can
stimulate creativity. And each user can choose the one which best fits his mood.

> Hopefully its fast enough. Seems fine. Good enough for controllers and
> basic/loop/pattern sequencing, maybe not for actually playing keys and
> pads and whatnot.

Okay, well, this Web UI idea would certainly be good in certain cases. And the
truth is that I'm not really convinced about the OpenGL idea, because it may
result in inconsistent appearance, with widgets style which do not match each other.

But this later problem could be solved with a simple audio-oriented UI toolkit,
which would render using OpenGL. A such toolkit could actually be very
lightweight. For instance you do not necessarily need to embed entire charsets.
A couple of characters (digits, "dB", etc..) could be sufficient in many cases.


More information about the Linux-audio-dev mailing list