Streams and large data in the browser

The Streams API in the browser – reading and writing data chunk by chunk instead of waiting for an entire file or response to land in memory. Useful for large downloads, uploads, log tailing, and progressive rendering.

The problem with buffering everything

response.text() or response.json() waits until the full body arrives. A 200MB export can blow the tab’s memory budget and block the UI while parsing. Streams let you process data as it arrives.

ReadableStream from fetch

const response = await fetch('/api/export.csv');
const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('n');
  buffer = lines.pop() ?? '';

  for (const line of lines) {
    if (line.trim()) handleRow(parseCsvLine(line));
  }
}

The line-splitting pattern handles chunks that cut mid-line. Keep a buffer for the incomplete tail until the next chunk arrives.

Pipe through transforms

Streams compose with pipeThrough and pipeTo. A transform stream maps chunks – decode bytes to text, parse JSON lines, filter rows.

function lineSplitter() {
  let buffer = '';
  return new TransformStream({
    transform(chunk, controller) {
      buffer += chunk;
      const parts = buffer.split('n');
      buffer = parts.pop() ?? '';
      for (const line of parts) {
        controller.enqueue(line);
      }
    },
    flush(controller) {
      if (buffer) controller.enqueue(buffer);
    },
  });
}

await fetch('/api/log')
  .then((r) => r.body)
  .then((body) =>
    body
      .pipeThrough(new TextDecoderStream())
      .pipeThrough(lineSplitter())
      .pipeTo(new WritableStream({
        write(line) {
          appendLogLine(line);
        },
      }))
  );

Backpressure

Streams have backpressure: a slow consumer signals the producer to pause. If you manually read with getReader(), respect desiredSize on writable sides. Piping handles most of this for you – prefer pipeTo over manual loops when you can.

Streaming uploads

async function uploadFile(file) {
  const stream = file.stream();
  await fetch('/api/upload', {
    method: 'POST',
    body: stream,
    headers: {
      'Content-Type': file.type || 'application/octet-stream',
      'Content-Length': String(file.size),
    },
    duplex: 'half',
  });
}

Streaming uploads need server support and correct CORS headers. Not every hosting stack accepts a raw stream body – test your endpoint.

Generating streams in JavaScript

function countStream(from, to) {
  return new ReadableStream({
    start(controller) {
      for (let n = from; n <= to; n++) {
        controller.enqueue(n);
      }
      controller.close();
    },
  });
}

for await (const n of countStream(1, 5)) {
  console.log(n);
}

for await...of works on async iterables from streams in modern browsers – handy for readable consumption code.

Streams plus workers

Heavy parsing can happen in a worker while the main thread reads chunks from the network. Post each chunk (or batch of lines) to the worker instead of the full payload. Combine with transferables when using binary formats.

When not to bother

• Small JSON API responses – response.json() is fine.
• Data that must be random-access (sort entire set before display) – you may still stream into an indexed structure, but plan for memory.
• Legacy browsers without stream support – feature-detect and fall back to buffered reads.

The next tutorial covers client-side security – XSS, CSRF awareness, and safe DOM APIs.