How local AI works in the browser with WebGPU and React

What actually runs on your machine

The phrase "local AI" gets used loosely enough that it is worth being exact about which piece of software is running where.

The model runs inside the builder — the React application at northerngo.com that you type your prompt into. When you switch on Local AI, a quantised language model is downloaded into your browser and executed on your graphics card. The Builder agent that writes your app's code is that model. Nothing about your prompt or the resulting source is sent anywhere.

The app you generate is a different thing. Generated apps are a single HTML document using the Tailwind CDN, with the NorthernGo SDK injected in a script tag. They do not carry a language model of their own. If a generated app calls window.NorthernGoAI.generate(prompt), that request goes over the network to the platform's model proxy, exactly as it would from any other web page. Local mode changes how your app is built, not how it runs for your users.

That distinction matters for the reason people usually ask about local AI in the first place: confidentiality of the code and the brief, not of the end users' data. If what you need is control over where your users' data ends up, that is a separate question answered by building GDPR-compliant apps in the EU.

The check that decides whether any of this is possible

WebGPU is the browser API that exposes the GPU for general computation. Everything else depends on it being present:

if (!navigator.gpu) {
  // No WebGPU: local mode cannot start at all.
  showMessage('Your browser or computer does not support WebGPU.');
  return;
}

In practice that means a desktop Chromium browser — Chrome or Edge. Safari and Firefox support has moved during 2025 and 2026, but the builder assumes a desktop Chromium engine and says so in the error message when generation fails. A dedicated GPU is not strictly required, but integrated graphics with little available memory will be slow enough that the cloud path is the better choice.

Mobile is technically possible and deliberately discouraged. The interface shows a warning before it starts a multi-gigabyte download over a mobile connection, and again before activating a cached model on a phone, because the RAM and battery cost is real.

The five models

Local mode is not a single model. You pick one in the Select AI model dialog, and the choice is a genuine trade-off between download size, speed and how well the model follows instructions.

• Qwen 3.5 (4B) — 3.0 GB, available on every plan including Free. The default. For small, clearly described pages, forms and simple changes — not a first CRM.
• Qwen 3.5 (9B) — 5.5 GB, Pro. The largest of the local models. Slightly better than 4B on longer instructions. Still far from the cloud model.
• Qwen 2.5 Coder (7B) — 4.8 GB, Pro. Code-tuned. Often the most reliable of the local set on HTML and JavaScript.
• Llama 3.1 (8B) — 5.1 GB, Pro. A general 8B model in the same class as the others — not a reasoning leap over the cloud.
• Ministral 3 Reasoning (3B) — 2.2 GB, Pro. Smallest and fastest to load. Weakest of the set; use when memory is tight.

The download happens once. After that the weights sit in the browser cache and activation takes seconds. The builder remembers which model you last used, so the next session skips the model picker entirely and starts it directly.

Free and Premium accounts get Qwen 3.5 (4B) with no limit on how many times you generate, which is the part worth internalising: the monthly credit allowance of 5, 25 or 50 generations applies to cloud generation only. Local generation is unmetered because it costs the platform nothing.

Wiring a multi-gigabyte engine into a React app

If you are building something similar yourself, two implementation details are worth stealing, because both were bugs before they were patterns.

Import the engine dynamically. The web-llm package should never be part of your initial bundle — a visitor who never touches local mode should never download it:

const { initLocalAI, generateLocalCode } = await import('../utils/localAI');

Hold the engine state outside the component tree. The activation control appears in two places in NorthernGo — a button in the navigation bar and a control in the prompt box — and both are mounted at the same time. When each held its own isLoading state, turning the engine on in one left the other showing it as off, and a fast double click could start the same 5 GB download twice. The fix is a module-level store with a single in-flight promise: a second call for the same model returns the promise that is already running, and a call for a different model while one is downloading is rejected outright rather than queued.

Loading itself reports progress through a callback, which is what drives the percentage in the interface:

import { CreateMLCEngine } from '@mlc-ai/web-llm';

const engine = await CreateMLCEngine('Qwen3.5-4B-q4f16_1-MLC', {
  initProgressCallback: (report) => {
    setProgress(Math.round(report.progress * 100));
    setStatusText(report.text);
  }
});

One more thing that is easy to miss: switching models requires unloading the previous one. Two sets of weights will not fit in video memory at the same time, and the second load fails in a way that is hard to read.

await engine.unload();

Streaming, and why the temperature is so low

Generation uses an OpenAI-shaped chat completion API, which means the code you write against it looks like ordinary model code:

const chunks = await engine.chat.completions.create({
  messages: [
    { role: 'system', content: systemContext },
    { role: 'user', content: prompt }
  ],
  temperature: 0.1,
  stream: true
});

let fullText = '';
for await (const chunk of chunks) {
  fullText += chunk.choices[0]?.delta?.content || '';
  onStream(fullText);
}

The temperature is 0.1 rather than a more usual 0.7 for a specific reason. These are small models being asked to obey a long and rigid format specification — emit only the inner HTML, no document wrapper, use Tailwind utility classes. Creativity at this step produces prose, apologies and half-finished markdown rather than better design. The design decisions come from the prompt, not from sampling randomness.

Speech and images use a separate stack. Text-to-speech and transcription run through @huggingface/transformers pipelines with device: 'webgpu', in Web Workers so the main thread stays responsive, and they are initialised in the background in parallel with the text model.

What still touches the network in local mode

Being precise here matters more than the marketing line, so: local mode is not the same as offline.

For a new build on Premium or Pro, the Architect step calls the cloud first to sketch the app's structure, and that blueprint is then handed to the local model. On the Free plan that step is skipped entirely. When you modify an existing app rather than build a new one, the Architect call is skipped for everyone — the local model receives the current body content and produces a targeted diff instead.

There is no LLM Critic rewrite on the local path. A syntax check blocks JavaScript that will not parse; it does not repair logic, and it is not a cloud round trip.

So: on Free, generation is fully local. On Premium and Pro, a description of what you asked for is sent to the cloud once per new build unless the Architect call fails, in which case the builder proceeds without it. If you need a guarantee that nothing leaves the machine, connecting Ollama for local models or building on the Free plan are the honest answers. The wider trade-off is covered in the comparison of local and cloud AI app builders.

Where local generation is genuinely worse

• Context window. The WebGPU models have a fixed and comparatively small window, often 4k to 8k tokens. A large existing app plus a full instruction set does not fit, which is why modifications are sent as diffs rather than whole documents.
• Instruction adherence. Smaller models sometimes ignore the diff format and return a whole new app instead. The builder detects this and retries once with a stricter prompt, but the retry costs another full generation of wall-clock time.
• Speed. A 9B model on a mid-range GPU is slower than the cloud path, sometimes considerably. The compensation is that you can run it a hundred times a day at no cost.
• First run. Between 2.5 GB and 6.2 GB has to arrive over your connection before anything happens at all.
• No mobile story. It works on a phone; you will regret it.

Troubleshooting

"Local generation failed" immediately, before any download. WebGPU is unavailable. Use desktop Chrome or Edge, and check that hardware acceleration is enabled in the browser's settings.

The download restarts every time. The model is cached in browser storage. Private or incognito windows, aggressive cache clearing and some enterprise policies discard it. The builder also keeps the chosen model id in localStorage — if storage is blocked entirely, activation still works but the model picker will reappear each session.

Generation fails partway through with a memory error. Video memory ran out. Switch to a smaller model, close other GPU-heavy tabs, and restart the browser to release any weights that were not unloaded cleanly.

"Another model is already being downloaded." Exactly what it says: a download for a different model is in flight. Wait for it to finish rather than reloading the page, which will only start over.

The result is a page describing the rules instead of your app. A small model has read the instruction block as the specification. Retrying with a shorter, more concrete prompt usually fixes it; so does moving to Qwen 2.5 Coder, which is markedly better at holding onto a format.