Live Generation | TanStack AI Docs

You want a video that plays while it generates. You also want to change the prompt mid-run. A finite video job stops with a file. generateLiveVideo() opens a session instead.

Call generateLiveVideo() on the server. It returns a connect payload: a model id, a prompt, and a token. The browser uses that payload to open the stream.

Experimental. The API can change. Live models bill per session-second while a GPU is held.

1. Mint a session on the server

Pick one adapter. They do not return a download URL.

import { generateLiveVideo } from '@tanstack/ai'
import { reactorVideo } from '@tanstack/ai-reactor'

const live = await generateLiveVideo({
  adapter: reactorVideo('helios'),
  prompt: 'A chef tosses noodles in a steel wok, flames leaping',
})

// Hand live.token, live.model, and live.prompt to the browser.

import { generateLiveVideo } from '@tanstack/ai'
import { reactorVideo } from '@tanstack/ai-reactor'

const live = await generateLiveVideo({
  adapter: reactorVideo('helios'),
  prompt: 'A chef tosses noodles in a steel wok, flames leaping',
})

// Hand live.token, live.model, and live.prompt to the browser.

import { generateLiveVideo } from '@tanstack/ai'
import { falLiveVideo } from '@tanstack/ai-fal'

const live = await generateLiveVideo({
  adapter: falLiveVideo('minimax/h3-max/director'),
  prompt: 'Live shopping stream: a host holds up a gold watch to camera',
})

// live.model is 'fal-ai/minimax-h3-max-director'. Hand it to the browser.

import { generateLiveVideo } from '@tanstack/ai'
import { falLiveVideo } from '@tanstack/ai-fal'

const live = await generateLiveVideo({
  adapter: falLiveVideo('minimax/h3-max/director'),
  prompt: 'Live shopping stream: a host holds up a gold watch to camera',
})

// live.model is 'fal-ai/minimax-h3-max-director'. Hand it to the browser.

Set REACTOR_API_KEY or FAL_KEY, or pass apiKey in the adapter config. Do not put the key in the JSON body.

A full app lives in examples/ts-react-media.

2. Connect in the browser

The server half is the same for every live adapter. The browser client is not.

Reactor

Install @reactor-team/js-sdk. Connect with the token. Helios uses set_sr_scale, then set_prompt, then start.

import { Reactor } from '@reactor-team/js-sdk'

const reactor = new Reactor({ modelName: live.model })
const video = document.querySelector('video')

reactor.on('trackReceived', (name, _track, stream) => {
  if (name !== 'main_video') return
  if (!video) return
  video.muted = true
  const attach = () => {
    video.srcObject = null
    video.srcObject = stream
    void video.play().catch((error: unknown) => {
      if (error instanceof DOMException && error.name === 'AbortError') return
    })
  }
  attach()
  for (const track of stream.getTracks()) {
    track.addEventListener('unmute', attach)
  }
})

await reactor.connect(live.token)
await reactor.sendCommand('set_sr_scale', { sr_scale: '2x' })
await reactor.sendCommand('set_prompt', { prompt: live.prompt })
await reactor.sendCommand('start', {})

import { Reactor } from '@reactor-team/js-sdk'

const reactor = new Reactor({ modelName: live.model })
const video = document.querySelector('video')

reactor.on('trackReceived', (name, _track, stream) => {
  if (name !== 'main_video') return
  if (!video) return
  video.muted = true
  const attach = () => {
    video.srcObject = null
    video.srcObject = stream
    void video.play().catch((error: unknown) => {
      if (error instanceof DOMException && error.name === 'AbortError') return
    })
  }
  attach()
  for (const track of stream.getTracks()) {
    track.addEventListener('unmute', attach)
  }
})

await reactor.connect(live.token)
await reactor.sendCommand('set_sr_scale', { sr_scale: '2x' })
await reactor.sendCommand('set_prompt', { prompt: live.prompt })
await reactor.sendCommand('start', {})

FastH3 uses enqueue only. LongLive uses set_shot then start.

Helios can also take a seed image. Pass a File from <input type="file">. Do not send base64.

const picker = document.querySelector('input[type="file"]')
if (!(picker instanceof HTMLInputElement)) {
  throw new Error('Pick a seed image')
}
const file = picker.files?.[0]
if (file === undefined) {
  throw new Error('Pick a seed image')
}
const image = await reactor.uploadFile(file)
await reactor.sendCommand('set_conditioning', { prompt: live.prompt, image })
await reactor.sendCommand('start', {})

const picker = document.querySelector('input[type="file"]')
if (!(picker instanceof HTMLInputElement)) {
  throw new Error('Pick a seed image')
}
const file = picker.files?.[0]
if (file === undefined) {
  throw new Error('Pick a seed image')
}
const image = await reactor.uploadFile(file)
await reactor.sendCommand('set_conditioning', { prompt: live.prompt, image })
await reactor.sendCommand('start', {})

A later set_prompt morphs the shot at the next chunk.

See the Reactor adapter for model ids.

fal H3 Max Director

Install @fal-ai/client@alpha. Keep FAL_KEY on the server. live.model is the WMA app id. Open WMA through a proxy that attaches the key. Do not send live.token as Key credentials.

The example proxy is src/routes/api.fal.proxy.ts in examples/ts-react-media. It attaches the key and forwards only to wma.fal.run (/ice, /session, /session/heartbeat) and Director /ice.

import { createFalClient } from '@fal-ai/client'
import { wma } from '@fal-ai/client/realtime'

const fal = createFalClient({ proxyUrl: '/api/fal/proxy' })
const video = document.querySelector('video')

const session = fal.realtime.open(wma(live.model), {
  receive: ['video', 'audio'],
  onError: (error) => {
    console.error(error)
  },
  onMedia: (stream) => {
    if (!video) return
    video.muted = true
    video.srcObject = stream
    void video.play().catch((error: unknown) => {
      if (error instanceof DOMException && error.name === 'AbortError') return
    })
  },
})

session.send({
  type: 'configure',
  prompt: live.prompt,
  prompt_version: 1,
  protocol_version: 1,
})

await session.ready

import { createFalClient } from '@fal-ai/client'
import { wma } from '@fal-ai/client/realtime'

const fal = createFalClient({ proxyUrl: '/api/fal/proxy' })
const video = document.querySelector('video')

const session = fal.realtime.open(wma(live.model), {
  receive: ['video', 'audio'],
  onError: (error) => {
    console.error(error)
  },
  onMedia: (stream) => {
    if (!video) return
    video.muted = true
    video.srcObject = stream
    void video.play().catch((error: unknown) => {
      if (error instanceof DOMException && error.name === 'AbortError') return
    })
  },
})

session.send({
  type: 'configure',
  prompt: live.prompt,
  prompt_version: 1,
  protocol_version: 1,
})

await session.ready

To steer, send { type: 'prompt', prompt, prompt_version } and increase prompt_version each time. To stop, send { type: 'stop' } and close the session.

Director bills a 60 second minimum. Resolution is 480p or 768p. See the fal adapter.

Models

Adapter	Id	What it does
reactorVideo()	helios	Interactive realtime video
reactorVideo()	fast-h3	Fast live clips on a live track
reactorVideo()	longlive-v2	Shot-based live clips
falLiveVideo()	minimax/h3-max/director	Steerable live stream over WMA

For Orbis and LingBot, use World Generation. For a file that finishes, use Video Generation.

What you have now

A server call that returns a connect payload. The browser opens the stream and steers it until you stop.