New Features

• Add cost tracking and cost-based rate limiting.
• Add namespaces: the ability to set up multiple granular experiments (A/B tests) for the same TensorZero function.
• Improve reasoning support for Anthropic (including adaptive thinking), Fireworks AI, SGLang, and Together AI.
• Allow users to whitelist automatic tool approvals for TensorZero Autopilot.
• Report provider errors when include_raw_response is enabled.
• Add include_aggregated_response to streaming inferences. When enabled, the final chunk includes an aggregated output aggregated_response that combines previous chunks.
• Allow users to kill ongoing evaluation runs from UI.
• Allow custom gateway bind addresses with the environment variable TENSORZERO_GATEWAY_BIND_ADDRESS.

& multiple under-the-hood and UI improvements (thanks @Nfemz @greg80303)!

New Features

• Support reasoning models from Groq, Mistral, and vLLM.
• Support multi-turn reasoning with Gemini and OpenAI-compatible models.
• Support embedding models from Together AI.
• Add configurable total_ms timeout to streaming inferences.
• Display charts with top-k evaluation results in the TensorZero Autopilot UI.
• Add "Ask Autopilot" buttons throughout the UI.
• Allow TensorZero Autopilot to edit your local configuration files.
• Return thought and unknown content blocks in the OpenAI-compatible endpoint (tensorzero_extra_content).

& multiple under-the-hood and UI improvements!

Bug Fixes

• Fix a regression in the aws_bedrock provider that affected long-term bearer API keys.
• Fix a horizontal overflow issue for tool calls and results in the inference detail UI page.

New Features

• Add YOLO Mode for TensorZero Autopilot.
• Add interruption feature for TensorZero Autopilot sessions.
• Add summary to the TensorZero Autopilot session table in the UI.

& multiple under-the-hood and UI improvements (thanks @pratikbuilds)!

Bug Fixes

• Fix a race condition in the TensorZero Autopilot UI that could disable the chat input.
• Increase timeouts for slow tool calls triggered by TensorZero Autopilot (e.g. evaluations).

& multiple under-the-hood and UI improvements!

New Features

• [Preview] TensorZero Autopilot — an automated AI engineer that analyzes LLM observability data, optimizes prompts and models, sets up evals, and runs A/B tests. Learn more → Join the waitlist →
• Support multi-turn reasoning for xAI (reasoning_content only).

& multiple under-the-hood and UI improvements!

New Features

• Add native support for provider tools (e.g. web search) to the Anthropic and GCP Vertex AI Anthropic model providers. Previously, clients had to use extra_body to handle these tools.
• Improve handling of reasoning content blocks when streaming with the OpenAI Responses API.
• Handle inferences with missing usage fields gracefully in the OpenAI model provider.
• Improve error handling across the UI.

& multiple under-the-hood and UI improvements!

Bug Fixes

• Fix a regression that triggered incorrect warnings about usage reporting for streaming inferences with Anthropic models.
• Fix a bug in the TensorZero Python SDK that discarded some request fields in certain multi-turn inferences with tools.

New Features

• Improve error handling across many areas: TensorZero UI, JSON deserialization, AWS providers, streaming inferences, timeouts, etc.
• Support Valkey (Redis) for improving performance of rate limiting checks (recommended at 100+ QPS).
• Support reasoning_effort for Gemini 3 models (mapped to thinkingLevel).
• Improve handling of Anthropic reasoning models in TensorZero JSON functions. Moving forward, json_mode = "strict" will use the beta structured outputs feature; json_mode = "on" still uses the legacy assistant message prefill.
• Improve handling of reasoning content in the OpenRouter and xAI model providers.
• Add extra_headers support for embedding models. (thanks @jonaylor89!)
• Support dynamic credentials for AWS Bedrock and AWS SageMaker model providers.

& multiple under-the-hood and UI improvements (thanks @ndoherty-xyz)!

New Features

• Support appending to arrays with extra_body using the /my_array/- notation.
• Handle cross-model thought signatures in GCP Vertex AI Gemini and Google AI Studio.

& multiple under-the-hood and UI improvements (thanks @ecalifornica!)

Bug Fixes

• Stop buffering raw_usage when streaming with the OpenAI-compatible inference endpoint; instead, emit raw_usage as soon as possible, just like in the native endpoint.
• Stop reporting zero usage in every chunk when streaming a cached inference; instead, report zero usage only in the final chunk, as expected.

New Features

• Support stream_options.include_usage for every model under the Azure provider.

& multiple under-the-hood and UI improvements!

New Features

• Add optional include_raw_usage parameter to inference requests. If enabled, the gateway returns the raw usage objects from model provider responses in addition to the normalized usage response field.
• Add optional --bind-address CLI flag to the gateway.
• Add optional description field to metrics in the configuration.
• Add option to fine-tune Fireworks models without automatic deployment.

& multiple under-the-hood and UI improvements (thanks @ecalifornica @achaljhawar @rguilmont)!