Benchmarking WebAssembly Builds: Unveiling Performance Insights

Dilla is a Design system into a universal WebAssembly package, executable through a simple declarative API, it caters to both server-side and headless rendering needs.

In the dynamic landscape of web development, using and optimizing WebAssembly (Wasm) builds is a meaningful endeavor to demonstrate the advantages of WASM. We choose to write our Engine with Rust because of it's maturity to create WebAssembly, for size and performance.

As the WebAssembly world is still nascent, we at Dilla have explored various approaches to build our Wasm final product. Now, it's time to measure performances within our specific context.

We measured performance in a JavaScript context, with Node for server side and Chrome/Firefox for the client side, across three different WASM build with:

• Wasm-bindgen
• Component model Jco
• Component model with Node WASI
• Extism

The Significance of Relative Differences

Before we delve into graphical and tabular results, it's vital to note that absolute times from benchmark tests may lack standalone significance. What truly matters is the relative difference between various times. We've strived to maintain conditions as close as possible during our tests to ensure the validity of comparisons.

Graphical Results: A Visual Journey

Let's kick off the exploration with some visually compelling graphics that bring our benchmark results to life:

Node execution

Note: Front bar is time without Node bootstrap, second bar include bootstrap time. Wasmtime run is given for reference.

Browser execution

Now, let's shift gears and delve into the details of how these graphics were made and how we interpret the results.

Test Protocol: Unveiling the Methodology

Our tests are done with Bootstrap 5 Design System on a Linux machine, devoid of active software or cron jobs, ensuring a pristine testing environment.

The focus of our performance measurements is on Dilla, encompassing two payloads that provide some real-world challenge and some very (very) large use case:

Normal Payload

• Size: 632 KB - Uncompressed JSON lines: 16,391

Large Payload

• Size: 1,9330 KB - Uncompressed JSON lines: 48,810

Note: A smaller payload ~200KB would be probably closer to real usage, but would be not enough to highlight relative performances.

Payload content

The payload include almost all components for Bootstrap 5 and all Dilla Render API features multiple times: @variables, @local_variables, @attached and @library. Duplicated for the bigger payload.

To know more about Dilla payload format, check the documentation on our Render API.

Tools

For Node tests, we leverage the powerful Hyperfine to measure performance including a full Node bootstrap.

To exclude Node bootstrap time we use node:perf_hooks.

All results are an average of 300 runs with 15 warm up, with σ lower as possible (except for Extism*).

Node files used for test can be seen in this repository, for each builder:

• Wasm-bindgen bench
• Component jco and Node WASI
• Extism

The command used is simply node --no-warnings _file__.mjs and for perf_hooks only the render phase is measured as seen in the bench.mjs files above.

Browser tests, on the other hand, are conducted manually on a clean install of Chrome and Firefox, with averages calculated from 30 values.

Tabular results: A Comprehensive Overview

The tabular results provide a detailed breakdown of Dilla WASM builds performance across various configurations.

Speed Results

For a normal payload, Wasmtime runtime is faster than Node with bootstrap, no surprise here. But without bootstrap time, Node is pretty efficient. For a large payload, differences tend to decrease.

Even with a wrapper, Component with Jco is faster than pure Node WASI, which is intriguing due to the memory allocation added by Jco. Waiting for a stable Node WASI might unveil its advantages.

Extism showcases a notable divergence in our tests. With a fluctuation of ±25 to 35ms, compared to ±1 to 7ms observed in others, Extism's behavior merits further exploration.

Firefox is, without surprise, outperforms Chrome with WebAssembly, demonstrating 17% to 28% faster performance during our tests!

Browser execution is around 25% slower than Node with full bootstrap, which shows the performance of SpiderMonkey, V8 is clearly behind for WebAssembly.

WASM Size

Size is the wasm file plus the JavaScript wrapper and module if any, minified and uglified. Wasm bindgen wrapper js is minified/uglified by hand, as it's not included in Wasm-bindgen cli as stated in the wasm-bindgen documentation.

WASM size is optimized as much as possible through common Rust build profile and wasm-opt. wasm-opt is the same for all 3 builds: wasm-opt -Os --low-memory-unused --enable-bulk-memory Difference being that Jco run it with transpile command when it's a manual step for Extism and WASM bindgen.

Dilla WASM builds size comparison in browser all accounted.

Component model Jco (wasm core + core2 + js wrapper)

• 664kb

Wasm Bindgen (wasm + js wrapper)

• 561kb

Extism (wasm + js lib)

• 714kb

Size Result

Wasm bindgen size is overall the best, in a server context, it may not matter much, but for browser client side it's crucial for us.

Fortunately, with WASM, we're talking only about the first load. Using our Dilla universal package allows caching the same WASM file across different websites, making the rendering akin to a static website!

Usage for Build and Code

Let's compare simplicity and usage of different build for browser usage (for node there's almost no difference).

Here is the build command and the minimal code we need to run Dilla with Node and Browser (without bundler for simplicity).

cargo component build --release
        npm install @bytecodealliance/jco binaryen
        npx jco transpile --name "bootstrap_5 "target/wasm32-wasi/release/my.wasm" --out-dir "."

import { render } from 'bootstrap_5.mjs'
        const result = render('{"@component": "alert", "message": "Hello!"}')

<script type="importmap">
          {
            "imports": {
              "@bytecodealliance/preview2-shim/cli": "./node_modules/@bytecodealliance/preview2-shim/lib/browser/cli.js",
              "@bytecodealliance/preview2-shim/filesystem": "./node_modules/@bytecodealliance/preview2-shim/lib/browser/filesystem.js",
              "@bytecodealliance/preview2-shim/io": "./node_modules/@bytecodealliance/preview2-shim/lib/browser/io.js",
              "@bytecodealliance/preview2-shim/random": "./node_modules/@bytecodealliance/preview2-shim/lib/browser/random.js"
            }
          }
        </script>
        const dilla = await import('bootstrap_5.mjs')
        let result = dilla.render('{"@component": "alert", "message": "Hello!"}')

import { readFile } from 'node:fs/promises'
        import { WASI } from 'wasi'
        const wasi = new WASI({
          version: 'preview1',
          args: ['', 'render', '{"@component": "alert", "message": "Hello!"}'],
        })
        const wasm = await WebAssembly.compile(await readFile('bootstrap_5.core.wasm'))
        const instance = await WebAssembly.instantiate(wasm, wasi.getImportObject())
        wasi.start(instance)

cargo build --target wasm32-unknown-unknown --release
        wasm-bindgen "bootstrap_5.wasm" --out-dir "." --out-name "bootstrap_5_bg.wasm" --target web
        wasm-opt -Os --low-memory-unused --enable-bulk-memory "bootstrap_5_bg.wasm" -o "bootstrap_5_bg_optim.wasm"

import { render } from 'bootstrap_5.js'
        const result = render({"@component": "alert", "message": "Hello!"})

const dilla = await import('bootstrap_5.js')
        await dilla.default()
        const result = dilla.render({"@component": "alert", "message": "Hello!"})

cargo build --target wasm32-wasi --release
        wasm-opt -Os --low-memory-unused --enable-bulk-memory "target/wasm32-wasi/release/my.wasm" -o "bootstrap_5.wasm"
        npm install @extism/extism

import createPlugin from '@extism/extism'
        const plugin = await createPlugin('bootstrap_5.wasm', { useWasi: true })
        const req = await plugin.call('render', new TextEncoder().encode('{"@component": "alert", "message": "Hello!"}'))
        let result = new TextDecoder().decode(req.buffer)
        await plugin.close()

import createPlugin from './node_modules/@extism/extism/dist/browser/mod.js'
        const plugin = await createPlugin('bootstrap_5.wasm', { useWasi: true })
        const req = await plugin.call('render', new TextEncoder().encode('{"@component": "alert", "message": "Hello!"}'))
        let result = new TextDecoder().decode(req.buffer)
        await plugin.close()

Usage results

Build steps are quite similar, each requiring an external library, CLI, or sub-command for JavaScript and/or Rust.

For usage, Wasm-bindgen and Component Jco for Node offer the most straightforward experience with their respective wrappers.

Node WASI needs a bit more but allows direct use of the WebAssembly JavaScript interface without wrapping it, which is advantageous.

Result Summary: Unveiling Patterns and Anomalies

Once upon a time in a JavaScript world

Wasm-bindgen and Component Model with Jco or native Node WASI exhibit close performances in terms of both speed and size.

A standout performer emerges in Component with Node WASI, potentially the optimal choice, eliminating the need for an additional wrapper. However, it's imperative to acknowledge the experimental nature of Node WASI. Our tests uncovered occasional crashes with Node 20 and a few with Node 21 and a limit for the payload around 2M. It's noteworthy that Node WASI commenced with Node 12, but updates have been gradual, with minimal progress since Node 19.

Component Jco emerges as a promising contender, although its relative immaturity compared to Wasm-bindgen The ongoing support for WASI preview 2 and the Component Model make it the clear future solution for us. But for now, potential breaking changes making it less suitable for production.

Wasm-bindgen stands out for now as a winner for both browser and Node usage, it's well documented and pretty simple to use. But there's some downside and blockers for us:

• Yet no WASI, issues: WASI support #3421, Align with the "C" ABI #3454
• Different wasm files for Browser and Node
• Support only JavaScript

Extism, while slightly slower and larger due to its generic approach seems behind, but...

Universality Unveiled: Extism Leading the Way

While our tests predominantly focus on JavaScript, we didn't yet addressed a fundamental aspect and the core reason for adopting WebAssembly in our Dilla product: Universality!

Extism not only meets but exceeds expectations, embodying the promises of WebAssembly and transforming our package into a truly universal solution. Its unmatched portability is a testament to its prowess.

Currently, Extism offers support for an extensive array of languages, including JavaScript (Web, Node, Deno and Bun!), Go, Java, .NET (supports C# and F#!), PHP, Python, Ruby, C, C++, Haskell, Elixir, OCaml, Zig and Rust.

We have successfully provided code snippets for utilizing our Extism build with targeted languages like PHP, Python, Go, Ruby, .NET, Java Additionally, this website employs PHP rendering for server-side builds of our live examples, as demonstrated on our components pages.

Is Extism the ultimate winner? Well, not so fast. Its reign continues until WASI and Component Model achieve stability. At that point, Extism leverage the Wasmtime runtime, but we hope the runtime itself will achieve the same universality, and it's backed by the Bytecode Alliance.

Wasmtime officially supports Python, Go, Ruby, .NET, C, C++, and Rust. However, it's still in the early stages, and we haven't yet successfully integrated our build with our targeted languages, unlike with Extism.

Conclusion: Navigating the WebAssembly Landscape

As of now, our strategic decisions revolve around ensuring robust JavaScript support for Dilla through Component Model Jco. Simultaneously, we designate Wasm-bindgen as the production-ready solution, combining efficiency and reliability.

On the server-side rendering front, Extism takes the lead, showcasing its unparalleled portability and universal application. We eagerly await the moment when Component Model achieves its universal target, potentially becoming the overarching solution.

The landscape of WebAssembly is dynamic, and our approach remains flexible as we anticipate further advancements and optimizations, but at present, our gaze is firmly set on Component Model and the Bytecode Alliance as the heralds of the future.

Stay tuned for the unfolding chapters in our journey at Dilla with WebAssembly.

Editor note: text written by a human and enhanced with AI

Further reading:

• The WebAssembly Component Model