Plugin test procedure

Plugins
1

Hello all,

As most of you know, numerous great plugins have been shared recently here in the forum that are not available in the Beta Store, nor in the “normal” Store. We want to change that and make the process of moving plugins forward more streamlined.

Now, with The mod-plugin-cookbook: using AI to create new plugins this problem can quickly escalate (and that would be a great signal).

To move them forward, we will need a lot of help from the community and, of course, from the developers. It is quite common that plugins get stuck because the developer can’t design a UI, not enough testing has been done, no one from the MOD Team gave a “stamp” to move it, etc. I personally have lots of interest in testing new plugins, but they should not get stuck in a limbo simply because I (or someone from our team) lacks the time to test them or even the skill to understand/use them. Documentation (both text and video) for the plugins and presets is also something that I really want to see improving A LOT in the near future. Again, I feel that some plugins don’t get the deserved love simply because someone along the way didn’t understand them, or even the majority of users can’t understand them. That’s a pity, especially when they can be of great use to lots of users.

I’ve been thinking for the last couple of days about what I personally should be checking during a test process. Obvious things are, for example: does it load? Does it crash the device? Does it input and output audio? Can we make assignments? So I thought in order to systematize the process, a spreadsheet could be handy. So I started working on it, but I feel that likely some plugins still could find a rabbit hole in the process and have some problem that we would not look into. On the other hand, with hopefully many more plugins, this process should be open to the community.

So I feel that the best way to kick it is to share the spreadsheet with all of you.

For now, I only opened it for comments, so you can give me suggestions on what is missing and I can input it there without starting to get messy. Maybe in the future I will open it for editing. I just need to understand what’s the best and safest process.

I divided the testing into 4 big areas:

  • Fuctionality
  • Audio
  • UI
  • Assignments

Do you think that we need more?

I see already a loophole there. For example, the OilCan MIDIrouter or the DrumGen Ultimate are MIDI plugins, so nothing is expected from the Audio part. The same with the Pluckface that is a CV plugin (this one I still haven’t tested myself). How can we adapt the spreadsheet so it fits any type of plugin?

Then I added the checks that need to be done for each section, with a final column for any required notes. The idea is that at the end of all tests, we get a pretty green line for the plugin, meaning that all musts are fine.

Any thoughts will be more than welcome!

My doubts at the moment are:

  • What should we test for each section?
  • Which sections do we need?
  • How to deal with different types of plugins (Audio, MIDI, CV, Generators, Utilities, etc)?

I’m happy to open this discussion with this great community (that makes me so proud :slight_smile: )

Cheers!

João

11 Likes
2

Hi João

Great initiative. I had a look at the spreadsheet - and my only suggestion would be to expand the possible responses from YES/NO (e.g Crash/Crash Not Crash) to maybe 4 levels

  • Pass
  • Mild Problems
  • Bad Problems
  • Fail

Thinking of the various plugins I have used which have caused problems in the past, they are:

ALO Looper (Memory Issues)

Calf Synth (Causes the Loopback option to disappear)

Distrho Sound Font Player (sorry I cant remember the name) - Only works on Midi Channel 0 and some sf2 files cause the player to stop working such that you have to delete it and then re-add it to the pedal board to reset it (totally unpredictable which .sf2s cause this behaviour

The ALO Looper’s problems only become visible after you’ve added 3 instances to the board - and then the memory usage keeps growing until the mod crashes (it also has a milder problem with the double clicking of a track on off - intended to reset a loop to re-record it, can rarely cause the other loops to reset also if the double click is very fast)
The Calf Synth must have a javascript problem because when you add it to the board, the midi loopback option no longer appears
The Distrho Sound Font Player’s problem does not affect the system or ui as a whole, but has this strange issue (.sf2 loading) that makes it problematic to use - though it has no consequences to the system as a whole. And the midi channel 0 only is not really a bug, but it does add friction to its use.
(Below is supposed to be a table, but its not rendering!)

| Plugin | System Safe | UI Safe | fit-for-purpose | Internal Bugs

|------|------|------|------|

| ALO Looper | Bad Problems | Pass | Pass | Mild Problems

| Calf Synth | Pass | Bad Problems | Pass | Pass

| Distro Sampler | Pass | Pass | Mild Problems| Bad Problems

I personally think “Mild Problems” should be allowed - I very happily used the ALO looper but was aware of its limitations (I have rewritten it for myself and hope to release it over the summer) - also the Calf Synth - as long as you set up the midi loopback before you add it to the board, you can get round it. Even the Distro Sampler getting uncooperative after certain sound fonts are added can be circumvented (you just have to take it off the board and bring it back on again)

Some final suggestions. For those of us amateur plugin authors, who maybe have fulltime jobs also, getting a plugin from personal use to general use (particularly writing the UI) takes time. Might there be a way to say to the mod team/community - I’ve done all I can - can you take it over please? (Obviously for open source plugins)

Also might there be a way for Alpha (or Gonzo) plugins to be incorporated into a special category of pedalboard sharing - such that we could share pedalboards with very early versions of plugins so that others could try them out in a context they were designed for?

Anyway, thanks for getting the ball rolling on developing a process for quicker beta-fication of new plugins!

3

Can you explain what you mean with “assignments” ?

4

I suppose they are binding or addressing, when you assign a parameter to a device knob.

1 Like
5

Hey @steve,

Although I get what you mean, I feel that adding more layers than “works or not” (or what you called “Pass” / “Fail”) would add ambiguity to the process. The idea here is to move the plugins to the stable shop (even those that aren’t in any shop), and that’s where we aim to start, because the developers are actively creating and interacting here. If they have “mild problems,” it’s okay to move them, but I think it’s best to send them to the beta shop until those problems are solved. We really need to avoid unpolished stuff on the official side of the platform. That would be a “call for problems”.

Your argumentation on particular plugins, I believe, is on its own a justification for why the “black & white” system is the best here. I take an example of what for you can be a mild problem, but for other users a bad one:

Imagine that, for whatever reason, the user does not have a controller with the MIDI channel 0 available. In this case, the plugin becomes completely useless.

And I could go on. Therefore, I guess that a “pretty green line” per plugin would be exactly what we want. If there’s some red point, it’s fine and maybe we leave it in beta for a while until the issue gets solved (if no one manages or attempts to solve it for a long time, it should be removed and eventually live only on a link here on the forum, imo).

Anyway, your suggestions for the table, I guess, can be good. Especially dividing “Functionality” into “System Safe”, “fit-for-purpose” and “Internal Bugs”

EDIT: one possible take-out on your first suggestion would be for the problem that I first mentioned to distinguished Audio, CV, MIDI and generator plugins, with some “non applicable” option. Readers: do you think that would be the best solution?

1 Like
6

I must say that I felt a need for, but I guess I never heard of a plugin with problems on that.

Basically, I mean if you can assign parameters on 1)directly on the device, 2)MIDI controllers and 3) Control Chain controllers

1 Like
7

That sounds great, thank you. :smiley:

According to the MIDI standard, there are only MIDI channels 1–16. You should stick to established general standards so that you can use the device with other devices (effectively). IMHO

1 Like
8

It’s a good extra check of course, but I wouldn’t think that any issues with this would be on the plugin side in the first place.

One thing we see with some beta plugins is that they can cause serious issues in the rest of the MOD-UI (I guess related to their own plugin UIs modifying the web app). So some additional tests, and review, here would be good.

9

:star_struck: Great initiative, been asking for streamlining of the publishing pipeline for a long time :smiley:

Getting more plugins publicly available AND easy to import and test is key

1 Like
10

Yes. That’s my point. I don’t really expect it to fail, but as it is a “must work” functionality, it’s worth going around all the parameters and trying to map them to different stuff.

Exactly, here is where I want/need more help on setting what/where we should look into.

To check this type of stuff, I had earlier the “Functionality” section, which, after the suggestion from @steve, I changed to “System safe”. I’ve already divided it here into two really important things: “Crashes the device” and “Crashes the WebGUI”. I can remember examples of the device crashing (which is a really big red) and others of only the WebGUI crashing (which could be more forgiven, but imo not for the stable shop…actually, while writing this I thought about other common issue that I will add there, crashing the device’s HMI (while still making sound, etc.)…something that I would say is in between crashing the device and crashing the WebGUI on a “how unusable”/red scale.

particularly on this, can you point me to some other checks that should be done more than “Crashes the WebGUI”? Thinking loud, I’m coming up with “removes or breaks anything from the WebGUI?”…what more?

11

I guess getting a stream of released plugins is one of the biggest strengths of the platform. Regardless of the hardware limitations, you can always have new and exciting plugins being released. That’s 1) like having a new pedal (without spending that much or even any money), and 2) refreshing tools for your music inspiration.

I think in the end of the day our goal is to keep inspired on making music. And I feel that a lot of music tech companies and even “geek musicians” forget a lot of times that the ultimate goal is to have tools to make music. Regardless of how “shinny” they are or how heavy or light in CPU they are or how “pretty” is the algorithm behind.

Lot’s of this new plugins that have been shared in the last year or so and are not even on the Beta store took me 5min to get something that really inspired me and gave me some music idea. It’s a pity if we let all of that sink.

12

My concern exactly!

If the A2 format proofs usable on the Dwarf, that’s an extra motivation.

Only a few devices give me this 2x2 routing to combine a vocal channel.
Creating my own amp models has been great fun as well.

I see examples of “Shadows in a box” plugins etc but I can’t be put to fiddling to get it to my Dwarf in some backdoor console way. I AM asemi-tech guy but when it comes to my music playing time, I like to try things, not try to get them work :smiley:

Still ,the Dwarf is one of the few devices I currently use and I hope to do so for a long time.
It’s product lifecycle already has to deal with the “small community” challenge, let’s not allow to make it stall due to the “lack of news/new items” challenge :wink:

1 Like