Nils‘ K1v in a browser as web assembly using Emscripten

Lately, I experimented a bit with VSTGUI again and the result is a running Kawai K1, directly in your web browser 😃

TL;DR if you want to try it: Click Here

🛑Needs a decent browser that supports web assembly! 🛑

If you want to know more about the implementation details, continue reading below.

Nils‘ K1v – VSTGUI, OpenAL, SDL2 – compiled as web assembly using Emscripten.

This idea evolved while I thought about how to release my other project, Heat Synthesizer as a native VST/AU including a UI. At the moment, it is an Android app with a small proxy VST for Windows only. The reason for not having a UI in the VST version is that I created it for Android using the native Android UI system in Java. Not an optimal solution to port it to different platforms.

To be able to port Heat Synthesizer to VST & AU, I was in need of a UI system that supports a wide range of platforms:

Android – Obviously I don’t want to drop support here so this is a must have
VST on Windows – I want to use my synthesizer directly in my DAW that I personally use, without the need to fire up my Android tablet to edit sounds
AU/VST on Mac – Would be beneficial if I were able to release it there, too

The Heat Synthesizer engine code compiles just fine for a wide range of platforms. I even had it running on an iPad already many years ago, just for testing purposes. But having a UI framework that covers all these platforms is not easy to find.

Actually, I recreated the UI in Unity3D some years ago. I thought this would be a good candidate to be more flexible with regards to targeting different platforms. Which is true for both iOS and Android, but Unity3D even nowadays lacks embedding support into runtime libraries so bringing it to VST and AU plugins wouldn’t be possible.

For the K1v, I’ve chosen a different strategy right from the start. Just in case you missed my development blog so far, K1v is created in C++ and the UI is made with VSTGUI. It’s a UI system by Steinberg, optimized for audio software. VSTGUI is open source and has become quite flexible throughout the years. It is no longer tightly bound any kind of existing VST SDK but also works with AU, VST3 or even in standalone mode.

What I came up with was the question if it was possible to create a fork of VSTGUI that supports Android. Knowing that I have to redo the Heat Synthesizer UI completely from scratch, my K1v project came in very handy here. K1v is a VSTGUI-based project that already compiles on a large variety of platforms (Win, Mac, Linux, both 32 and 64 bit each) that I could play around with.

K1v in Standalone Mode

Obviously, the first step was to create a standalone app of Nils‘ K1v. An Android app would be a „Standalone“ app, not a plugin. That step was pretty much straightforward and was finished after one evening.

The internal structure of K1v is that there is a generic „K1Plugin“ and a „K1Editor“ and these two are then wrapped for the different plugin platforms such as VST2, VST3 (unreleased yet) and AU.

I created an executable that opens a VSTGUI-based standalone window, embedded the K1v editor and created the K1v plugin instance.

Audio Output? MIDI?

To test the standalone app, I needed an audio output and an audio processing loop. Luckily, I solved this years ago as part of the Heat Synthesizer project. There is a simple audio output interface class with a lot of different implementations:

OpenSLES – Used on Android
Jack – Used for Samsung Soundcamp on Android. Unfortunately they stopped development and dropped support, but this implementation might come in handy on Linux in the future
OpenAL – I used this for testing purposes as the SDK is very small and I already knew it from game development
PortAudio – I added this some years ago to have lower latency on Windows
CoreAudio – Used to test on Mac & iOS

I opted for PortAudio and OpenAL for now as I have been able to test them directly on Windows.

As for MIDI, there is just no support included. To test that everything works, I injected a Note ON event on Note C3 at startup, should be enough. I didn’t plan to release a standalone version anyway so no need to implement MIDI here.

Emscripten & WebAssembly

You may ask why there is a web version now even though I kept writing about Android?

For me, Emscripten is an intermediate step. I didn’t want to deploy to my Android tablet over and over again to test my progress so I thought that Emscripten as a first challenge would be a good candidate to start with.

As I plan to keep most of the code in C++ anyway, both platforms are more similar than expected. Android an Emscripten are both „based“ on Linux (more or less), they have OpenGL ES as possible renderer and as I use CMake as a build system on all platforms differences are very minor during build.

Additionally, I’m not sure if an Android version of K1v is really useful. the role of Heat Synthesizer is different here, as it contains lots of online functionality (preset exchange) and can connect to a VST proxy for proper DAW integration, but as K1v is a VST already that doesn’t make much sense.
Releasing it for the web on the other hand is a nice challenge and furthermore something that didn’t exist before, at least I’m not aware of any existing VSTGUI to Emscripten ports.

Porting VSTGUI to Emscripten

While my K1v code compiled more or less directly, a lot of effort was needed to get VSTGUI up and running.

Everything I write here is based on VSTGUI 4.9, to be more specific, the develop branch at 2020.06.24 commit dc95f6a2. This is the version I used to develop the UI for Nils‘ K1v. I did some bug fixes and fixed Linux compilation errors at this time, but other than that, this is the state I’m currently working with / what the Emscripten version is based on.

The VSTGUI core code (everything that is platform independent) only had smaller compilation problems in one or two cases, but the largest issue was that I didn’t have anything for this new platform: No frame, no window, no run loop etc.

I decided to start with the Linux version. The Linux version is based on Cairo for rendering, uses X11/XCB for input handling and GDK for the main loop in Standalone builds.

Cairo for rendering

I knew that Cairo can be compiled in a way to use OpenGL ES for rendering, which both Android and Emscripten support. So i decided to fork Cairo and compile it for Emscripten using CMake.

This brought a lot of dependencies: zlib, libpng, pixman, libpixman.

More projects that I had to fork. Some of them already had CMake build scripts, which was great, for others, I created CMake build scripts from scratch. It took some time but worked eventually.

Although the web assembly based executable linked fine, I still had a lot of work to do. I didn’t solve the standalone part yet, i.e. the window handling and mouse & keyboard input.

I created some stub classes to be able to test it. At last, I heard the K1v preset IA-1 Ahh playing Note C3 in a browser window, yeah 😎

Using SDL2 to have a render target for Cairo

The next problem I had to solve was to define some sort of „window“ as that is what a standalone app in VSTGUI is based on. The problem is that you do not have any sort of window in Emscripten, as you’re running in a web browser. You get a canvas that you can paint on, that’s it.

Luckily, the Emscripten team already solved this problem by porting the SDL library to Emscripten. This was great news as the SDL library also supports Android.

I spent quite some time connecting an SDL „window“ to Cairo. I wanted to give hardware acceleration to Cairo so I compiled Cairo with OpenGL ES support and created a GL Surface in SDL. Unfortunately, I had to drop this idea for Emscripten as only a subset of OpenGL ES 2.0 is implemented and Cairo makes use of these missing features (for example pbuffers).

So instead of using the GL based backend, Cairo runs in software at the moment. Nevertheless, I was blown away when I saw parts of my UI for the first time:

Font Support

As you might have noticed, there are no texts in the UI. The reason for this is that the Linux build of VSTGUI uses fontconfig to enumerate the available system fonts at startup. I had to comment out that whole code block as there is no fontconfig support in Emscripten and adding this library doesn’t make sense as there are no „system fonts“ available anyway. So instead of using fontconfig, I did the following:

I created a virtual file system for Emscripten at compile time with a folder named „fonts/“ that contains all fonts that I need for K1v. These are not many and the resulting data blob is around 1Mib in size only.
At load time, this font list is built by enumerating through all files in this folder. Then, I load the files with FreeType to query for the family name and style name to finally rebuild the original fonts list that VSTGUI uses. This worked out pretty well and is completely customizable as the available fonts depend on what you put into the virtual filesystem. No hardcoded hacks were necessary.

Web assembly version of K1v with libcairo rendering fonts

Input Handling

As I had the SDL library, adding input handling was pretty straightforward. I use the SDL for mouse, keyboard & window events. I also use the SDL for timers.

Missing Features

The project is highly experimental and in a very early stage. VSTGUI support lacks the following:

file selector is not available
no alert boxes
no support window styles as there is no real window
no drag & drop
no tooltips
no clipboard support

The web version of Nils‘ K1v doesn’t currently support:

State is not preserved, if you reload the page, you’re going to have factory patches again, no matter what you edited before
MIDI – The only possible input is the virtual keyboard on the web page
Load or Save of files due to missing VSTGUI file selector

Most of the above could be implemented easily. Another thing worth mentioning is that I don’t currently use AudioWorklets as it seems to be quite cumbersome for now to get them running in Emscripten. The audio output is done by OpenAL and everything is single threaded. You can expect some audio pops & clicks here or there, it depends on your browser & computer, mostly noticeable when large UI parts are redrawn. Comments about how it works for you would be nice.

Future Work

After finishing the port to Emscripten, its time for some cleanup work, obviously. After I will have finished cleaning up the VSTGUI parts, I’m going to make my VSTGUI fork open source. I might decide to do a pull request, maybe Steinberg is interested to integrate official Emscripten support? Who knows.

To summarize, I had to create implementations for the following VSTGUI interfaces to make VSTGUI work with Emscripten:

VSTGUI::IPlatformTimer – SDL timers
VSTGUI::IPlatformFrame – SDL <=> Cairo frame
VSTGUI::Standalone::Platform::IWindow – SDL Window
For standalone mode, an SDL application which drives VSTGUI::Standalone::Detail::Application
Some groundwork such as an SDL run loop to drive redraws & inputs.

These are all great preparation steps to have an Android port of Heat Synthesizer. I’m going to continue this journey with the K1v project though, as I want to proof that my concept works before doing a complete UI redesign of Heat Synthesizer with VSTGUI.

I’m confident that the Android version of the SDL/libcairo combination will support hardware accelerated rendering, which is even more important on mobile. As Android has full OpenGL ES 2 support, this should work fine.

I’m going to post updates on this topic. Leave a comment and tell me what you think 👍

Nils‘ K1v: Version 1.14 released – Multi support!

I’ve just released version 1.14 of Nils‘ K1v – The Kawai K1 emulation VSTi/AU.

Biggest change is that it now also supports Multi presets. I added the 32 factory Multis and all Multis from the ROM cards, resulting in 384 Multi presets in total.
As a sum, Nils‘ K1v now contains 1352 presets.

Of course, Multis can be edited, too. To add all of the new features, I modified the UI slightly. The Source Mute buttons have been moved to the Single Edit page, making space for the new Buttons MULTI and SYSTEM.

I’ve been able to fit four Multi sections on one page, arrow buttons allow to switch between sections 1-4 and 5-8.

Multis work as they do on the Kawai K1 hardware. Up to eight Singles can be combined in various ways to create either rich layered sounds or multiple zones on the keyboard.

The only difference between a K1 and K1v is that in Poly Mode VR (meaning variable), the polyphony is unlimited, while it was still limited to 16 or 8 voices in total on a K1 hardware.

The SYSTEM page is a new page that allows to change global parameters, basically it’s a copy of the K1 Midi Settings and supports specifying a Midi receive channel (previous versions always ran in Omni mode) and to disable Program Change processing, Sysex & more.

Global settings are saved as part of the DAW project.

Besides the improvements as pointed out above, some additional things have been added, such as cursor key preset navigation (Hallo Jürgen!) and of course, bugs have been fixed, too. The full change log can be found below.

Download the new version from the K1v product page.

Changes

[Fix] Wrong number of presets (832 instead of 968) were reported to plugin hosts
[Fix] Midi notes could get stuck in some cases
[Fix] Rounding issue in fixed key note number drop down menu that caused note B0 to be displayed although note C1 was selected
[Imp] K1v now supports Multi presets. All factory Multi presets have been added for all factory banks, making up a total of 384 Multis, or 1352 presets in total
[Imp] K1v now has a settings page to allow to specify the Midi receive channel, disable/enable receive & send of SysEx and more
[Imp] Midi receive channel can now also be modified by sending an Omni-Mode-Off control change from a specific Midi channel.
[Imp] Added ability to navigate through presets with cursor keys
[Imp] Code optimizations to reduce overall CPU usage

Nils‘ K1v: Version 1.13 released

I just released Nils‘ K1v version 1.13.

The most noticeable change is that I have added all 200 PHm Pop Synth Module presets to the plugin, resulting in a total of 968 Singles.
As the list of Singles became even larger, I reworked the play mode section to make Single selection easier. There is now a dropdown to allow to select a bank directly and there are additional navigation buttons.

Another interesting change for some people might be one accuracy fix: Key Scaling =>Envelope Time didn’t modify Decay but only Attack, causing some Singles to sound incorrect.

More details & all fixes and changes can be found in the change log excerpt below. The new version is available for all platforms and can be obtained from the Download Page

Changes

[Fix] Drop down menus were hard to read on Linux because of the font being too tiny
[Accuracy] Key Scaling => Envelope Time modulation did not affect Decay but only Attack, causing some Single to sound incorrect when compared to the K1 hardware.
This change fixes the Single ‚KillDa Mix‘ and others, mostly Pianos.
[Imp] All 200 PHm presets are now part of the plugin, causing it to have 968 total built in presets.
[Imp] Single selection screen has been reworked, for easier navigation it now includes a dropdown to select the Bank and has more navigation buttons.
[Imp] Plugin Logo has been reworked to match the K1 hardware logo. Furthermore, the „KAWAI“ logo has been integrated as Kawai gave official permission for the plugin.

Nils‘ K1v: Version 1.12 released

I just released Nils‘ K1v at version 1.12. It contains two bug fixes and one important improvement, as I mentioned earlier in my development blog: Velocity Curves now very closely match those of the Kawai K1 hardware.

The release is available for all platforms and can be obtained from the Download Page

Changes

[Fix] One-shot waveforms (drums & others) were played looped
[Fix] Mac AU version crash when saving the plugin state (100% crash in Digital Performer 10)
[Imp] Velocity Curves are now properly implemented and very closely match those of the K1 hardware. If you are interested in details, feel free to visit the K1 development blog

Kawai K1: 100% accurate velocity curves – provided by the firmware ROM

Some users reported that the velocity response of my K1v plugin is quite different compared to the K1 hardware. And yes, that is probably the area with the biggest difference between the K1v and the K1.

As I have pointed out in an earlier article, tracking down the velocity curves is very difficult, as there is no way to get the raw velocity data. What you will capture on the audio out is always the envelope volume – modulated by the velocity curve. Getting the raw velocity data is impossible.

A different approach – using the K1m Firmware ROM

While sitting here and thinking about how to solve the unsolvable problem, I had an idea. The K1 needs to have the curves stored somewhere. I used my PHm ROM that I have dumped earlier and loaded it as raw data into my wave editor.

It didn’t took very long until I found interesting data tables that looked similar to the velocity curve graphs in the manual. Actually, there a plenty of data tables in there. Not only velocity, I also found the key scaling curves and lots of other tables. I still need to figure out what they are for, but the result will be a much better precision in the K1 emulation.

Velocity Curves (length 64) & KS Curves (length 128) in Kawai K1 firmware ROM

The velocity curves have a length of 64 bytes each and have a range of 0 to 127, as you’d expect for velocity values. I extracted them from the ROM and used them in the K1v, but unfortunately they did not really match the K1 output.

Some things that I observed:

Velocity Curve 1, that is supposed to be linear, is not linear. Velocity Curve 3 is linear although it should not.
In my recordings, for most curves, velocity values above 100 do not make any difference anymore. But the curves have proper values, why aren’t they used?

Data Analysis

Lets first have a look at what the manual says VS what is in the data:

Velocity Curves – as stated in the K1 manual

Velocity Curves – as part of the K1 firmware ROM

What can be seen immediately is that all curves in the ROM are less exponential than required.

Furthermore, not the whole velocity range is used, at least not when modulating envelope volume. Take a look at the picture below, this is a K1 recording of Velocity Curve 1 with all velocity values from 1 to 127.

Velocity Curve 1 – Velocity Levels 1-127

As you can see, envelope volume is peaking way before velocity maximum. To be precise, the maximum envelope level is reached already at velocity 102. I made the gap in the recording on purpose by inserting a quarter after velocity 100 so I had to count less bars to find the values 😊.
The same applies to all other curves. Either the incoming velocity values are fed through some other table before the curve is applied, or afterwards. By testing different values and their result, I was able to verify that there must be another remapping before the velocity curves are applied.

Implementation details

It took me many hours, a lot of guessing, thinking & testing before I found the correct table in the firmware ROM. It turned out that it is a table with a length of 64, it looks like this:

Velocity remapping before being fed into the different velocity curve tables

This curve explains why very high velocity values do not make any difference anymore, they are cut off. Furthermore, the whole curve makes everything a little bit more exponential.

I was pleased with the result: I had steps in envelope volume values at the same velocity values than my K1m. The but came quickly as the values didn’t match the K1m.

Intermediate result of K1v Velocity Curve 7

To make the result more exponential, I had to search for a table with a specific size: The envelope level has a range of 0 to 100, so a table with 101 values had to be the correct one.

As I had worked on this topic for such a long time, my source code with all kinds of tables grew more and more 😁 And now definitely needs some cleanup.

Firmware ROM tables converted to source code

Luckily, I found the correct table within one hour or so. It is a 16 bit table, it has a length of 101 as expected and makes the curves identical to the K1m.

Once more, Velocity Curve 7 as being output by the K1v, now 100% accurate:

Final result, K1v Velocity Curve 7 – identical to hardware K1m

You might see that there are very small differences due to rounding errors, but these shouldn’t really matter.

Two more pictures, a comparison of all velocity curves in all its glory 👍

Pending work

As I have confidence about what tables make up correct velocity curves, I need to implement proper modulation levels. So far, everything you see is at Vel => Env Mod +50, i.e. 100%. Special care needs to be taken when implementing the modulation strength as I currently have the impression that a value of +25 does not mean a modulation strength of 50%, there might be another remapping table being inserted to do this. Negative strengths are another topic that might be interesting.

Of course, once I’m done there will be another post to announce the release of the update.

Other useful data tables in the firmware ROM?

There are some other tables in the ROM, both 8 bit and 16 bit with data that looks interesting. If I can figure out what they are used for, I can improve the emulation a lot.

A 16 bit curve, length 101. Values from 1536 to 0 (at center) to 1536. Most probably modulation depth remapping

Another 16 bit curve with length 101, value range 0 – 4000. Most probably vibrato depth + others

For the latter curve, it looks very similar to one of my own curves that I had figured out by analyzing wave data.

Conclusion

If I ever go to emulate another synthesizer, putting more effort into ROM analysis instead of recording hours of data is the better way to go 😊 Stay tuned & thanks for reading!