Thinking in Streams

Most code assumes you have all your data before you start: an array in memory, a full response body, a complete file. Streams are for the (very common) case where you don't — where data shows up a piece at a time.

Think about an LLM typing out a response, a multi-gigabyte log file, rows from a database cursor, or bytes off a network socket. Waiting for the last piece before you do anything means more memory, more latency, and a worse experience. Streams let you start working on the first chunk while the rest is still on its way.

The Web Streams API is built into modern browsers, Node, Deno, and Bun. You really only need three nouns:

• ReadableStream — a source you pull values out of (a file, a fetch body, an array you wrap).
• WritableStream — a sink you push values into (a file, the console, a socket).
• TransformStream — a step in the middle: values go in, (different) values come out.

And one verb that connects them: .pipeThrough() for transforms, .pipeTo() for a final sink.

// A stream is values arriving over time — not all at once.
const lines = fetch("/big.log")
  .then(r => r.body)               // ReadableStream<Uint8Array>
  .pipeThrough(new TextDecoderStream())
  .pipeThrough(splitLines());      // ReadableStream<string>

That's the whole shape: source → transform → transform → sink. Everything in this library is just a nicely-typed source, transform, or collector you can drop into that chain.

The thing that surprises people: Web Streams are pull-based. A source doesn't fire data at you as fast as it can. The consumer asks for the next chunk when it's ready, and that "I'm ready" signal travels back up the whole pipeline.

This is backpressure, and you get it for free. If your slow database write can only keep up with 100 rows/sec, the fetch at the top of the pipe automatically slows down to match. No manual buffering, no unbounded memory growth, no drain events to babysit.

The most useful trick is to picture a stream as an array stretched out across time. You can't index into it or call .length — you only ever see one element at the "now" line — but the operations you already know still apply:

• map transforms each value as it passes.
• filter drops values that don't match.
• reduce / scan fold values into an accumulator.
• take / drop slice by position.

That's why the marble diagrams in the API reference are worth a look: each one shows values flowing left-to-right along a timeline, so you can see what a transform does instead of decoding a type signature.

Here's the quick-start, annotated. arrayStream turns a plain array into a source, the transforms run in order, and collect drains everything back into an array at the end.

import { arrayStream, collect, filter, map } from "@withremyinc/stream";

const output = await collect(
  arrayStream([1, 2, 3, 4])
    .pipeThrough(map(n => n * 2))      // 2, 4, 6, 8
    .pipeThrough(filter(n => n > 4)),  // 6, 8
);
// [6, 8]

The streaming parsers are where this gets genuinely useful. Feed in text as it arrives and get structured events out immediately — ideal for LLM token streams:

import { parseJSON } from "@withremyinc/stream";

// Tokens from an LLM arrive a few characters at a time.
// parseJSON emits structured events as soon as it can — no waiting
// for the closing brace.
modelStream
  .pipeThrough(parseJSON({ emitPartialStrings: true }))
  .pipeTo(renderAsItStreams());

Streams are a tool, not a religion. If your data is small and already in memory, a plain array and Array.prototype.map is simpler and faster to read. Reach for streams when at least one of these is true:

• The data is large enough that holding it all in memory is a problem.
• The data arrives over time and you want to act on early chunks.
• You need backpressure between a fast producer and a slow consumer.

The Web Streams API gives you the primitives; it does not give you map, filter, merge, or a tolerant streaming JSON/XML parser. That's the gap @withremyinc/stream fills: small, composable, well-typed helpers that snap onto the standard pipeThrough/pipeTo chain.

Ready to build something? Head to the API reference — every helper has a marble diagram and a runnable example. Working with an AI agent? The agent skill teaches it to use these correctly.