Web-assembly and WebGL usage in modguis (C/C++ for modguis)

[falkTX]

Want to open a topic to discuss something potentially interesting for plugins on the MOD platform.
I have been experimenting with building and running native desktop programs on the browser, with the biggest project being the whole Cardinal on the browser. See https://cardinal.kx.studio/ for that.

The way this works is a mix of web-assembly (compiled through emscripten) and graphics over webgl.
It works on the 3 major browsers, with old Safari/WebKit being a bit behind with the support for SIMD, but if we are only dealing with graphics we dont need to care about fast audio optimizations.

I see a few clear benefits of this approach:

• We can use the same desktop UI directly as modgui on MOD
• Graphics are typically very fast to render, as it is using the GPU
• Developers that dislike web stuff (HTML/CSS/JS) can skip it altogether and only deal with C/C++

For the moment I have been tweaking my own little DPF framework to make this as seamless as possible, but it is very much still work in progress.
And now I am wondering if anyone has interest on testing this out, or even porting/developing plugins with this in mind.

For a good first test, let’s begin with Wolf-Shaper, with is a “Waveshaping distortion plugin with spline-based graph”.
If you have a MOD Dwarf, you can install it using this one-liner on the command-line:

curl https://falktx.com/data/wasm/wolf-shaper.lv2-dwarf.tar.gz | base64 | curl -F 'package=@-' http://192.168.51.1/sdk/install

Find it at the end of the distortion tab.

There are a few quirks to its usage, as I am trying to find a good way to allow to interact with the plugin while still allowing to move it around. Initially I had a border around the GUI, but that ends up wasting space and looking ugly. Instead I made it so that when you single-click on the GUI (and the web gui zooms and centers to it), the GUI finally receives focus allowing you to tweak knobs and the spline graph.

A few other small oddities might be present, this is experimental after all.
Try with while keeping these in mind:

• if plugin GUI does not have focus, click away on the pedalboard once, then back to the plugin GUI
• plugin GUI is quite big, you might need to unpin the plugin bar in order to reach the bottom knobs
• for the spline editor, right-click on empty area to add new point; right-click on point for edit options
• the GUI wont work on Safari/WebKit for now, I know why, will fix it later
• under Chrome the mouse pointer lock doesnt always work well, under investigation
• works best under Firefox
• might not work correctly on high-dpi displays, currently untested
• developer console shows lots of debug info, expected and normal
• oversample is a knob, even though it doesnt look like it; big cpu usage if enabled

if you try it, let me know how it works for you!

oh and some screenshots too, because not everyone will be able to try it, but they will want to see.

[dreamer]

This is super cool! I think this will also make it a lot more interesting to port “real” plugin UI to MOD.

Personally I think it definitely will help to be able to focus on a single UI toolkit, rather than having to reproduce things in html/css/js.
Very much looking forward to what people will come up with using this!

To what extend do you think non DPF-based plugins can make use of a similar strategy?
I suppose their respective frameworks will need to be adapted to make use of this.

[AndyCap]

Very interesting, I will give that plugin a test out after a couple of days of drinking and eating.

[falkTX]

@dreamer

To what extend do you think non DPF-based plugins can make use of a similar strategy?

As you guess it most likely will depend on specific frameworks. JUCE had some initial work though from external developers, I have seen demos but seemed very initial stages.
The biggest issue for JUCE is how the UI is tied to the DSP side, not being able to decouple the 2.
For DPF I did a 2 step build where DSP is built with “native” toolchain and UI is built afterwards as 2nd step while still keeping the same environment and files.
Will need to investigate JUCE later, but I have heard the devs have DSP/UI separation in mind for future JUCE things… maybe we wait for that?

Since DPF uses pugl, some barebones/raw LV2 plugin can likely be made to work as well with the custom pugl backend.

[falkTX]

A few more tests for web-assembly and modguis, just by the screenshot would this be a yay or a nay?
Cardinal full GUI on the pedalboard:

the alternative would be opening a new dedicated browser tab/window.

[dreamer]

This is really cool!

I’m only worried that, since it’s a fixed limited size “window”, the real estate might be too uncomfortable to use.
While one should not make super huge and complicated patches inside MOD, opening the module browser may be too constrained in size to navigate easily.

Maybe it’s possible to have a “best of both” and be able to “launch” the UI into a separate browser tab when needed?

[edit: oh wait, it is resizable? or is that just a remnant of the plugin resize handle?]

[falkTX]

@dreamer

Maybe it’s possible to have a “best of both” and be able to “launch” the UI into a separate browser tab when needed?

What you do think about a full screen mode? So the pedalboard would still show the real cardinal rack canvas, perhaps smaller than it is now, but have a quick access button to make the GUI bigger.

@dreamer

[edit: oh wait, it is resizable? or is that just a remnant of the plugin resize handle?]

It could be made resizable, but I think that would be too weird

[dreamer]

Agreed to both, actually

“full screen mode” would then remain in the mod-ui view, except cardinal would temporarily “take over” and present itself as an overlay?
That could work, and advantage is that you can stay in the single window and not end up having a bunch of tabs open at the same time and have to go back and forth.

One question: in the current cardinal one can enable/disable the audio/cv i/o so you can make connections to/from the plugin. How are you thinking this would work here?

[falkTX]

@dreamer

“full screen mode” would then remain in the mod-ui view, except cardinal would temporarily “take over” and present itself as an overlay?

no overlay, that would be a bit too complex. the easiest and for getting the most usable space available is regular fullscreen.
what would be the point of the overlay? to allow to close it more easily…?

@dreamer

One question: in the current cardinal one can enable/disable the audio/cv i/o so you can make connections to/from the plugin. How are you thinking this would work here?

I would reserve this in-canvas view for the “mini” version of cardinal, because the intention is to have something that eventually could be moved to stable store. the amount of modules on the non-mini variants has surpassed 1000, as you know, so it is much more difficult to ensure everything works smoothly for those many modules.
on a mini version we would only have a few select modules, and as such 8 audio + 10 cv ports kinda doesnt make sense in my opinion. I am thinking 2 audio, midi and 5 cv are a good set of ports to go with.

[falkTX]

Continuing the adventures, the workflow of click-to-focus was too awkward so I have changed that now.
The bypass switch was also missing, which gave me an idea.
What if we use a window-like top-bar with the bypass switch on it? Then the “window title” space can be for the plugin name. It reminds me of the Gnome Desktop

Opinions welcome, what do you people think?

Made the wolf-shaper GUI smaller, which I think is better now as it leaves more room/space for other stuff.
Interaction is not so awkward anymore, though it takes some getting used to not being able to drag GUIs from non-control elements (with this design only the “title” bar can be grabbed for moving)

I fixed the issues preventing usage under Safari/WebKit, so likely can just push this to beta…
but need to see with someone else from the MOD team first if this makes sense to ever push to stable store eventually.

[falkTX]

Just updated the test link for those wanting to try, with some fixes regarding high-dpi and adjusting the size a bit.
This is an odd one to package, we will need to figure out the best way to deal with wasm binary assets…
For the moment Dwarf users with a Linux or macOS system can install wolf-shaper as described on the first post:

curl https://falktx.com/data/wasm/wolf-shaper.lv2-dwarf.tar.gz | base64 | curl -F 'package=@-' http://192.168.51.1/sdk/install

One cool little extra we get by default, if there are 2 web browsers open with the MOD web gui, the state of the wolf-shaper is kept in perfect sync!

[AndyCap]

I am having some difficulties in safari with this version.

The warning about the mouse cursor being hidden is displayed at the top which moves the mod GUI down, even more annoying if you are zoomed in this unzooms it!

[falkTX]

yeah safari makes this super weird to use…
the mouse pointer thing is to allow to keep mouse drag while it moves outside of the GUI bounds (otherwise the GUI stops receiving events)

maybe there is a way to ask a permission and keep it always on? they surely must realize this behaviour is quite obnoxious…

[AndyCap]

I have had a look to see if I can find anything in Safari to turn this off, the one way I can see is to just not use safari.

On an iPad over wifi it works perfectly!

[Steve_Lee]

You might want to check out the awesome https://stackblitz.com/
Amazing stuff. Currently a full nodejs web dev env in a browser but maybe more could be added to make a cool env for linux plugins?

Steve

[Austin73]

Now that would be cool.

For people like me with pretty much zero coding skills we could actually get on board and do some of the hard work porting plugins for the more tech savvy to mark our work and see if it makes the grade.

When I first looked into the Dwarf I must admit I thought any LV2 Plugins could be loaded on it. And if there was a possibility to finally do this I think sales would pick up very quickly.

Really love the stuff you are trying Falk but unfortunately I’m windows only so can check out at present. If that does change I’m happy to help in anyway.

[dreamer]

This is quite literally the opposite.

Personally I’m happy to not have to replicate work done for the desktop to re-implement in html/css/js.

[AndyCap]

@falkTX Did you think more about the “full screen” idea. For me it would be great if you could expand to a full GUI of the plugin in a new tab/window.

[falkTX]

the fullscreen-on-separate-tab is a bit complex to do, not sure if possible for the mini variant.
work for the mini variant benefits the other non-mini ones, where the remote control is being improved. things feel quite snappy now.

I am starting to get things working, finally.
Caught a small issue in mod-host along the way, so it is not useful to share builds just now until we have a 1.12.2 update.

but can still leave some pictures I guess…
the initial selection of modules have begun, plaits is one of the modules that works already.

I found a way to fake the IO animations, so the GUI should in theory show the same behaviour as the DSP/plugin side. This includes the scope

These are super low-resolution in terms of “audio” being generated, but just enough to give visual feedback.

[falkTX]

5 CV ports are enough for a mini version, right?
The monitoring of audio and CV works, even with its super low resolution on the GUI side.
Transport state can be sent too, with only MIDI remaining (but I am not sure if that is really needed…)