getting whisper to load in 2 seconds on locked-down work chrome

The symptom: works here, fails there, then fails everywhere

Our recorder runs Whisper in the browser so meeting audio never leaves the device. For a month it exhibited the worst kind of bug: the speech model loaded instantly on some Chrome profiles, hung forever on others — same machine, same build, same network — and occasionally froze the entire laptop hard enough to blame the hardware. Managed work profiles (the browsers consultants and employees actually have) failed most reliably, which for a product aimed at exactly those users was existential.

Every obvious hypothesis failed in sequence. It wasn't the GPU: a native onnxruntime control run constructed the same model in 47–272 ms on the 'degraded' machine. It wasn't the model files: byte-identical assets loaded fine in one profile and never in the next. It wasn't the network: the downloads completed, then the hang happened after them, inside session creation, with no error, no rejection, and no timeout — the promise just never settled.

Root cause 1: unset numThreads can hang session creation forever

onnxruntime-web's WASM execution provider spins up an emscripten pthread pool. With numThreads unset, the default pool sizing inside a Web Worker can deadlock InferenceSession.create() — it never resolves and never rejects (the ort#26858 family). Worse, the wedged construct's spin-locks can peg every core, which is how a JavaScript library convinced us for weeks that the machine itself was dying.

The fix is one line: set numThreads explicitly before the first session. We use min(4, max(1, hardwareConcurrency / 2)). With that set, the same construct that hung forever completes in ~1.6 seconds.

Root cause 2: the default optimization level breaks q8 models

With threads fixed, quantized Whisper (q8) died with a new error: "TransposeDQWeightsForMatMulNBits — Missing required scale". That reads like an unsupported-operator problem; it's actually a known Level-2 QDQ fusion bug in the graph optimizer (the ort#28306 family) present in the onnxruntime-web build we pin. Setting session options to graphOptimizationLevel: "basic" skips the buggy fusion pass entirely — the error vanishes, and as a bonus session creation gets faster. We measured no meaningful inference-speed loss for this model class.

The general lesson: on a pinned runtime, optimizer passes are part of your compatibility surface. When a quantized model fails with a fusion-sounding error, try turning the optimizer down before blaming the model.

Root cause 3: let the bundler pick your WASM files and it will pick wrong

onnxruntime-web ships several WASM builds (threaded/non-threaded, JSEP/asyncify). Passing wasmPaths as a directory string lets the runtime and bundler negotiate which pair to load — and under some bundler/version combinations they negotiate a mismatched or suboptimal pair. We pin the exact .mjs/.wasm pair with the object form of wasmPaths, self-hosted and same-origin, which also removes a CDN fetch that corporate content filters love to intercept silently.

Why not WebGPU? Because Chrome's shader cache betrays you

WebGPU inference is genuinely faster once warm. But 'warm' is a trap in Chrome: compiled shaders live in a Dawn cache that is roughly 6 MB, LRU, and shared across ALL origins — any shader-heavy site evicts yours — and it is invalidated by every Chrome update. On managed enterprise profiles WebGPU is frequently unavailable outright. So the cold-compile cost (minutes for a model like Whisper) is not a one-time cost; it is a cost your users pay again on a schedule you don't control. A product cannot assume a warm GPU cache in Chrome.

Our resolution: CPU/WASM is the engine — about 2 seconds to ready on any profile, 7–13× realtime for whisper-base q8, which comfortably covers live transcription — and WebGPU is demoted to an opt-in flag. The fastest path is worthless if it doesn't load; the path that always loads turns out to be fast enough.

Keeping live transcription honest on slow hardware

A live transcript on CPU needs a plan for the day inference falls behind the meeting. Ours has three parts. First, a streaming loop with LocalAgreement-2 commit semantics: text is only committed once two successive decodes agree on it, which kills the flickering-rewrite problem. Second, an RTF governor: if the un-decoded backlog exceeds 40 seconds, we deliberately skip ahead and keep only the most recent audio, so the live view stays near real time instead of drifting minutes behind — and we count every skipped second. Third, always-backfill: after the meeting stops, if anything was skipped (or the live pass produced nothing), a full re-transcription of the saved recording repairs the transcript. The live view is allowed to be approximate; the saved transcript is not.

The debugging method that actually worked

What cracked the month-long mystery wasn't a profiler; it was experimental hygiene. An isolated-Chrome lab (scripted Chrome with throwaway profiles) made 'cold profile' a reproducible input instead of an anecdote. A native onnxruntime control run separated 'machine problem' from 'browser-stack problem' in one afternoon. Binary-searching the pipeline (fetch → cache → worker init → session create → first decode) with per-stage logging found where 'forever' actually happened. And a ?debug=1 overlay in the product itself turned user machines we could never touch into instruments that report VAD state, ring depth, and decode counts in a screenshot.

The meta-lesson: every earlier verdict we'd formed under confounded conditions — 'the model is broken', 'the machine is degraded' — was wrong, and each had cost days. Verdicts formed while two variables moved at once aren't conclusions; they're folklore.

• numThreads: set it explicitly in workers — min(4, cores/2) works well
• graphOptimizationLevel: "basic" dodges Level-2 QDQ fusion bugs on q8 models
• wasmPaths: pin the exact .mjs/.wasm pair (object form), self-hosted, same-origin
• Treat Chrome's Dawn shader cache as hostile: 6 MB, LRU, cross-origin, reset by updates
• Governor + backfill: bound live lag, repair the saved transcript afterwards
• Isolate variables like it's a lab, because it is one

frequently asked questions

Which onnxruntime-web version does this apply to?

We pin a 1.26-dev build (with transformers.js 4.x). The numThreads hang class and the Level-2 fusion bug are tracked upstream (ort#26858, ort#28306 families) and fixed or mitigated in newer releases — but the configuration advice (explicit threads, basic optimization for q8, pinned wasmPaths) is cheap insurance on any version.

How fast is CPU-only Whisper in the browser, really?

On our hardware, whisper-base q8 constructs in ~1.6 s and transcribes at roughly 7–13× realtime via the WASM execution provider with 2–4 threads. That's ample for live meeting transcription with a streaming window, and it holds on managed work profiles where WebGPU isn't available at all.

Is this the product pitch in disguise?

The product (a private, on-device meeting recorder at nowrecording.com) is why we hit these bugs, and it's free to try without an account — but everything above is reproducible with transformers.js and onnxruntime-web alone, no product required.