Thinking in Streams

Every chat app streams text. Tokens arrive, you append them, and the reply types itself out. Users expect that now.

Then the model calls a tool, and the screen freezes. Tool arguments are JSON, and JSON.parse can't read half a document. So the app holds everything until the last brace arrives, and the user watches a spinner while the model writes a 200-line file they could have been reading.

The same thing happens whenever you ask a model for structure: props for a component, a list of search queries, a plan with steps. The more you ask for, the longer the user waits to see any of it.

parseJSON reads JSON one chunk at a time and reports each key and value as soon as it can. Pipe that into jsonToJSObject and you get the object so far, rebuilt after every change, including strings the model is still writing.

import { arrayStream, jsonToJSObject, parseJSON } from "@withremyinc/stream";

// Tool-call arguments, in the pieces a model sends them.
const toolArgs = arrayStream([
  '{"path":"src/Bu',
  'tton.tsx","contents":"export function',
  ' Button() {',
  ' return <button />; }"}',
]);

const snapshots = toolArgs
  .pipeThrough(parseJSON({ emitPartialStrings: true }))
  .pipeThrough(jsonToJSObject());

for await (const args of snapshots) render(args);
// {}
// { path: "src/Bu" }
// { path: "src/Button.tsx" }
// { path: "src/Button.tsx", contents: "export function" }
// { path: "src/Button.tsx", contents: "export function Button() {" }
// …

Your UI can show src/Button.tsx after the second chunk and start showing the file after the third.

jsonToJSObject hands you the same object each time and updates it in place, which is why it's cheap to emit on every change. Copy it before you keep it. React, for one, skips the re-render if you set state to an object it already has.

In generative UI the model decides what to show as well as what to say. It picks a component and fills in its props, and your app renders the result: a weather card, a flight list, a chart.

That output is structured, so it has the same problem as a tool call. Without a streaming parser the card appears all at once, after the model has finished every field. With one, the card mounts as soon as the model names it and fills in while the rest arrives.

A JSON spec

The common format is a JSON object with a component name and its props, sent as a tool call or as structured output.

import { jsonToJSObject, parseJSON } from "@withremyinc/stream";

const registry = { WeatherCard, FlightList, Chart };

// The model answers with { "component": "WeatherCard", "props": { ... } }.
const specs = reply
  .pipeThrough(parseJSON({ emitPartialStrings: true }))
  .pipeThrough(jsonToJSObject());

for await (const spec of specs) setSpec(structuredClone(spec));

function GeneratedUI({ spec }) {
  // "Weath" isn't in the registry, so nothing mounts until the name is complete.
  const Component = registry[spec.component];
  return Component ? <Component {...spec.props} /> : null;
}

A few things decide whether this feels good:

• Components get partial props. A five-day forecast may have two days so far, and the last one may have half a summary. Build components that render whatever they have and show placeholders for the rest. Most of the work in generative UI is here, not in the parsing.
• Key order is display order. The model writes keys in sequence, so put component before props, and short fields before long ones, so the layout appears before the paragraphs. OpenAI's Structured Outputs writes keys in the order your schema lists them.
• Some strings shouldn't render half-written. The registry lookup protects component names. For URLs and enum values, check against what's allowed before you use them, or drop emitPartialStrings to receive strings only once they're complete.

Components inside the reply

Other apps let the model write tags straight into its answer, like <weather city="Lisbon" />, so prose and components arrive in one stream. parseXML reads the reply as a fragment and hands you text and elements in the order they appear.

import { parseXML } from "@withremyinc/stream";

const parts = reply.pipeThrough(parseXML({ textMode: "delta" }));

for await (const part of parts) {
  if (part.type === "onText") appendText(part.value);
  if (part.type === "onElementBegin") mount(part.name, part.attributes);
}

// Chunks: 'Lisbon looks warm this week. <weath'
//         'er city="Lisbon" days="5" /> Pack li'
//         'ght.'
//
// onText         "Lisbon looks warm this week. "
// onElementBegin weather [city="Lisbon", days="5"]
// onText         " Pack li"
// onText         "ght."

The weather tag is split across the first two chunks and still arrives as one event. Attributes come together once the tag closes, so a component never mounts with half an attribute. If a tag's body is data, such as a chart's JSON, list it in foreignTags and its body comes through as text instead of being parsed as markup.

Models wrap JSON in prose and Markdown fences. They leave trailing commas and write comments. Some prompts ask for XML tags instead, and some replies open with a frontmatter header.

Each of those has a transform that goes in front of the parser. extractDelimiter streams the body of a fenced block, extractXML pulls out the tags you allow, and extractFrontmatter splits a header from the text after it.

// The reply is prose with a ```json block somewhere in the middle.
const plan = replyText
  .pipeThrough(extractDelimiter({ allowLanguages: ["json"] }))
  .pipeThrough(parseJSON({ emitPartialStrings: true }))
  .pipeThrough(jsonToJSObject());

The parsers also accept what JSON.parse rejects. Comments and trailing commas parse without complaint. Real mistakes become onError events, and parsing continues: {"a":1,,"b":2} reports two errors and still produces { a: 1, b: 2 }. One stray comma doesn't cost you the whole reply.

You could build all of this on callbacks or async generators. We built on the Web Streams API because model output already arrives as one. fetch returns a ReadableStream, and the same code runs in browsers, Node, Deno, Bun, and Cloudflare Workers with no polyfill.

const response = await fetch("/api/chat", { method: "POST", body });
if (!response.body) throw new Error("Response has no body");

const args = response.body
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(parseJSON({ emitPartialStrings: true }))
  .pipeThrough(jsonToJSObject());

There are three types. A ReadableStream produces values, a WritableStream consumes them, and a TransformStream sits between and turns one into the other. You connect them with pipeThrough and end the chain with pipeTo, a collector, or a for await loop. Most functions in this library return a TransformStream.

A chain built this way gets two things for free. The first is backpressure. Each stage asks for the next value only when it's ready, so if your UI renders more slowly than tokens arrive, reading slows down instead of buffering without limit. The second is cancellation. When the user clicks Stop and you cancel the stream you're reading, the cancel travels back up to fetch, which closes the connection, and the server can stop generating.

Backpressure and cancellation only work while the chain is unbroken. Read a stream by hand and write the values into a new one, and you've taken the job over yourself. Usually something gets dropped: an error that never reaches the UI, or a request that keeps running after the user leaves.

The agent skill turns this into rules a coding agent can follow and check its own work against.

You can't index a stream or ask for its length, because most of it hasn't arrived. Everything else you do with arrays still works, one element at a time.

import { arrayStream, collect, filter, map } from "@withremyinc/stream";

const output = await collect(
  arrayStream([1, 2, 3, 4])
    .pipeThrough(map(n => n * 2))      // 2, 4, 6, 8
    .pipeThrough(filter(n => n > 4)),  // 6, 8
);
// [6, 8]

The API reference draws each function as a marble diagram, values moving left to right along a timeline, which is often faster to read than the type signature.

If the data is small and you already have all of it, use an array. Streams add moving parts, and they're worth it when users should see output before it's finished, when the data won't fit in memory, or when a fast producer feeds a slow consumer.