Our /benchmark page is not marketing copy — it is generated from a harness (apps/benchmark-comparison) that installs both frameworks from the public npm registry and measures the same ten workloads on the same machine, in the same headless terminal. Every number on that page comes out of comparison.json. That makes the harness itself worth engineering, and it just got its biggest upgrade so far.

What this note covers

  • The checks that keep benchmark results comparable and reproducible.
  • What the current data reveals about strengths, bottlenecks, and uncertainty.
  • The exact commands for running a fast local smoke test or a full release run.

What the suite measures

The suite keeps each workload and terminal size fixed, then records how the render loop, memory footprint, startup path, and event handling behave. The interactive benchmark page turns those raw measurements into a workload-by-workload comparison, with plain-language descriptions of what each metric means.

Signal Question it answers
Frame time and FPS How quickly can the UI update while work is happening?
Input and scroll latency How responsive is an interactive terminal surface?
RSS and heap How much memory does the process retain under load?
Startup and import time How quickly can an application be ready to render?

The goal is not a single headline score. It is a useful map: select a workload that resembles your application, inspect the metric definition, and decide whether the trade-off matters for your users.

What changed in the harness (v2)

Medians instead of means. Headline timings are now medians across up to 60 in-process samples per phase, which shrugs off GC pauses and scheduler noise. The full distribution — mean, median, p95, stddev and coefficient of variation — is stored alongside every result, so “how noisy is this number?” is always answerable.

Repeated fresh-process runs. Every framework×workload pair now runs in --runs (default 3) separate processes. The orchestrator snapshots each raw run to disk, then publishes the element-wise median plus a stability block: cross-run CV% for frame time, fps and RSS. A metric that is not reproducible is now visible as such instead of silently averaged into a lie.

Noise-aware verdicts. A winner is only declared when the frame-time gap exceeds max(5%, 2× cross-run CV). Smaller gaps are reported as Even. This kills the classic benchmarking failure where run-to-run jitter manufactures a “2% faster” headline.

Parity guard. If the two frameworks ever end up with >10% different node counts for the same workload — API drift quietly shrinking one side’s workload — the comparison flags it instead of comparing apples to oranges.

History + regression diff. Every aggregation archives comparison.json under results/history/<timestamp>/, and pnpm bench:diff walks the two newest archives and flags per-framework, per-metric adverse moves beyond ±10%. Benchmarks that only report the present are half-blind; this makes the harness trend-aware.

A real developer loop. --quick halves sample counts, --app= filters workloads, --framework= picks one side, and the orchestrator warns when the machine’s load average exceeds its core count. A full run is for release data; a two-minute smoke run is for while you code.

What the current numbers already tell us

Full-run verdicts were stable across three repeats (bun 1.4.2, darwin 25.6.0, Apple M4, headless 120×40). The interesting part is not who wins overall — it is where each framework is structurally strong:

  • BetterTUI leads on deep trees (3.0× faster dirty frames at 5.1k nodes), memory on large workloads (roughly half the RSS of OpenTUI on 10k-row lists), module import time (~2–3× faster), and on building at all: past ~16k renderables OpenTUI hits a hard native cap and cannot construct the workload, which we report as a structured failure rather than hiding it.
  • OpenTUI leads on small frames, flat lists and — by a wide margin — scrolling: our 2k-row scroll workload frames at ~28× the cost of OpenTUI’s. That is the single largest gap in the dataset, and it is ours to close.

Where we could improve: the framework

The benchmark exists to aim work, so here is what the data says BetterTUI should get better at, in priority order:

  1. ScrollBox scrolling — the 28× scroll gap is the top offender. A viewport that re-renders all children instead of clipping is the likely shape of the problem.
  2. Small-frame overhead — trivial one-node workloads run 2–4× slower than OpenTUI’s. That constant factor is what users feel in snappy UIs.
  3. Flat-list dirty frames — 1.7× on 10k-row lists suggests per-node dirty propagation could batch better on wide, shallow trees.

Where we could improve: the harness itself

v2 fixes the statistics, but the measurement surface still has known limits we plan to remove:

  • Real PTY measurement. Today both frameworks render through their own test renderers. Measuring through a real PTY would capture the true cost of terminal writes and ANSI diffing — the part users actually wait on.
  • Bytes written per frame. Diff renderers live or die on how little they emit. Today’s “terminal throughput” assumes full-cell writes; counting actual output bytes would measure diff efficiency directly.
  • Adaptive repetition. Instead of a fixed --runs, repeat until the cross-run CV drops under a target, spending more time exactly where runs disagree.
  • Stronger statistics. Bootstrap confidence intervals (and eventually a rank-sum test) instead of ratio thresholds, per metric.
  • CI + alerts. Nightly full runs in CI, with bench:diff failures opening issues automatically.
  • Cross-platform matrix. One M4 Mac is one point in a large space — Linux, Windows and budget hardware will reorder some verdicts.
  • Framework-idiomatic workloads. Raw Box/Text primitives are the fair common denominator, but real apps use each framework’s higher layers (BetterTUI’s React adapter, OpenTUI’s solid bindings). Idiomatic-vs-primitive deltas will quantify what the adapters cost.
  • Realistic input. Paste bursts, key repeats and typing cadences instead of a single synthetic keypress.
  • Memory soak. A ten-minute leak window and per-burst peak tracking (the peak metric landed in v2; the soak is next), so “doesn’t leak” becomes a measured claim instead of a hope.
  • Cold vs warm startup. Module import is warm-cache today; a cold-cache split will say what first-run UX really costs.

Try it

Start with the quick run while you are changing code, then use the full run when you need numbers suitable for publication or a regression comparison.

Terminal window
pnpm --filter @bettertui/benchmark-comparison install
pnpm --filter @bettertui/benchmark-comparison bench:quick   # smoke run
pnpm --filter @bettertui/benchmark-comparison bench         # full 3-run release data
pnpm --filter @bettertui/benchmark-comparison compare       # aggregate + archive
pnpm --filter @bettertui/benchmark-comparison bench:diff    # trend vs previous archive

Published numbers on the /benchmark page will be refreshed from a full v2 run; expect small absolute shifts (means → medians) and honest Even labels where gaps live inside the noise. Visit the live benchmark comparison to explore the latest results and see which workloads support input or scroll latency measurements.