LLM Gateway | TanStack AI Docs

LLM Gateway is an open-source AI gateway that routes one OpenAI-compatible endpoint to hundreds of models across many providers — with automatic provider selection, fallback, usage analytics, and cost tracking. Use the hosted gateway at api.llmgateway.io or point the adapter at your own self-hosted deployment.

Installation

npm i @tanstack/ai-llmgateway

npm i @tanstack/ai-llmgateway

Basic Usage

typescript

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("gpt-5.6-terra"),
  messages: [{ role: "user", content: "Hello!" }],
});

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("gpt-5.6-terra"),
  messages: [{ role: "user", content: "Hello!" }],
});

llmGatewayText reads your API key from the LLM_GATEWAY_API_KEY environment variable. Use createLLMGatewayText to pass it explicitly.

Configuration

typescript

import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText(
  "gpt-5.6-terra",
  process.env.LLM_GATEWAY_API_KEY!,
  {
    baseURL: "https://api.llmgateway.io/v1", // Optional — set for self-hosted deployments
  },
);

import { createLLMGatewayText } from "@tanstack/ai-llmgateway";

const adapter = createLLMGatewayText(
  "gpt-5.6-terra",
  process.env.LLM_GATEWAY_API_KEY!,
  {
    baseURL: "https://api.llmgateway.io/v1", // Optional — set for self-hosted deployments
  },
);

LLM Gateway is open source and self-hostable; point baseURL at your own deployment to keep the same adapter surface.

Need a header on each request, or a log of each model call? This adapter supports wrapFetch.

Available Models

Any model listed at llmgateway.io/models works — pass its id as the model name. A bare model id lets the gateway route to the best available provider; prefix it with provider/ to pin routing to a specific provider:

text

model: "gpt-5.6-terra"          // gateway picks the provider
model: "claude-sonnet-5"        // gateway picks the provider
model: "moonshot/kimi-k3"       // always routed to Moonshot
model: "fireworks/kimi-k3"      // always routed to Fireworks

model: "gpt-5.6-terra"          // gateway picks the provider
model: "claude-sonnet-5"        // gateway picks the provider
model: "moonshot/kimi-k3"       // always routed to Moonshot
model: "fireworks/kimi-k3"      // always routed to Fireworks

A curated set of flagship models (see LLMGATEWAY_CHAT_MODELS) additionally carries per-model type metadata — input modalities and provider options — with editor autocomplete. Uncurated ids still work and fall back to text-only input with the generic options.

adapter.inputModalities lists the input kinds of the selected model. It is undefined for a model that this package does not know. See Check what a model accepts.

Example: Chat Completion

typescript

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
  });

  return toServerSentEventsResponse(stream);
}

import { chat, toServerSentEventsResponse } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
  });

  return toServerSentEventsResponse(stream);
}

Example: With Tools

typescript

import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}

import { chat, toServerSentEventsResponse, toolDefinition } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";
import { z } from "zod";

const getWeatherDef = toolDefinition({
  name: "get_weather",
  description: "Get the current weather",
  inputSchema: z.object({
    location: z.string(),
  }),
});

const getWeather = getWeatherDef.server(async ({ location }) => {
  return { temperature: 72, conditions: "sunny" };
});

export async function POST(request: Request) {
  const { messages } = await request.json();

  const stream = chat({
    adapter: llmGatewayText("gpt-5.6-terra"),
    messages,
    tools: [getWeather],
  });

  return toServerSentEventsResponse(stream);
}

Model Options

The gateway accepts the standard Chat Completions parameters and forwards them to the routed provider (parameters a provider doesn't support are stripped server-side). Sampling parameters live in modelOptions:

typescript

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
  },
  reasoning: "high",
});

import { chat } from "@tanstack/ai";
import { llmGatewayText } from "@tanstack/ai-llmgateway";

const stream = chat({
  adapter: llmGatewayText("kimi-k3"),
  messages: [{ role: "user", content: "Hello!" }],
  modelOptions: {
    temperature: 0.7,
    max_completion_tokens: 4096,
  },
  reasoning: "high",
});

Reasoning

reasoning goes out as reasoning_effort. The types take the levels of each model, and no budgetTokens:

kimi-k3, glm-5.2, deepseek-v4-pro, qwen3.7-max: off to max.
gpt-5.6-terra: off, low to max. gpt-5.5, gpt-5.4-mini: off, low to xhigh.
claude-opus-5, claude-sonnet-5, claude-haiku-4-5, minimax-m2.5: low to max.
gemini-pro-latest, grok-4-5: low to high. gemini-3.6-flash: minimal to high.

Reasoning models stream their thinking as reasoning_content deltas, which the adapter surfaces as AG-UI REASONING_* events.

Summarization

typescript

import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  stream: false,
});

console.log(result.summary);

import { summarize } from "@tanstack/ai";
import { llmGatewaySummarize } from "@tanstack/ai-llmgateway";

const result = await summarize({
  adapter: llmGatewaySummarize("gpt-5.4-mini"),
  text: "Long article text...",
  stream: false,
});

console.log(result.summary);

Environment Variables

Set your API key in environment variables:

LLM_GATEWAY_API_KEY=llmgtwy_your-api-key

LLM_GATEWAY_API_KEY=llmgtwy_your-api-key

Get an API key from the LLM Gateway dashboard.