7 best AI observability tools for production LLM apps

AI observability tools help explain why a large language model (LLM) app is slow, expensive, or returning incorrect answers in production. Gateways record cost, latency, and provider attempts without application tracing code. Tracing and evaluation tools capture the prompts, tool calls, and agent steps behind a response and help assess its quality. Many products combine these capabilities.

Copy link to headingWhat is AI observability?

AI observability combines production metrics, traces, and evaluations for LLM applications. It extends application performance monitoring (APM) with model usage, token costs, prompt and response context, provider reliability, and response-quality scores.

A request can return HTTP 200 even when the answer is incorrect, earlier provider attempts failed, or the token cost exceeded expectations. Each problem needs different evidence:

• Gateway metrics record latency, time to first token (TTFT), token usage, cost, and provider attempts for requests routed through a gateway.

• Application traces connect model calls with retrieval, tool execution, and other steps. They can capture prompts and responses when content recording is enabled.

• Evaluations, often called evals, assess responses through code-based checks, LLM-as-judge scoring, and human review. Repeated evaluations help detect regressions.

A tracing and evaluation platform can identify slow model calls and quality issues. AI Gateway instrumentation records routing decisions and upstream attempts that an application-level trace may not expose.

Copy link to headingSeven AI observability tools at a glance

The tables compare each tool's coverage, AI SDK integration, free allowance, and paid plans.

The allowances use different units. Check each provider’s website for the latest pricing.

Copy link to headingCompare integrations and trace compatibility

OpenTelemetry standardizes how traces are structured and transmitted. To display model names, token counts, and tool calls, a backend must also recognize their attributes. Some backends map multiple schemas automatically. Others accept spans but omit fields from their AI-specific views.

Send a test trace through the integration and check that the backend displays the fields you need. Read setup guides for supported destinations on the AI SDK observability integration directory.

Copy link to headingWhat gateway observability adds

An AI gateway routes requests to model providers and can manage retries, fallback models, credentials, and billing. Vercel AI Gateway Observability records usage and performance for requests sent through the gateway.

Application-level agent tracing follows the steps within an agent run. Grouping several runs into a session requires a session identifier or equivalent configuration. The agent observability guide explains how to follow model calls, tools, and intermediate results.

Vercel AI Gateway Trace Drains omit prompt and completion text. Request metadata can still contain identifiers or other sensitive information, so review the exported fields and who can access them.

Copy link to headingLatency by model and provider

An overall latency average can hide a regression. If more traffic moves to a faster model, the average can improve while another model becomes slower.

AI Gateway's request summaries show P75, the 75th percentile, for duration and TTFT by project and API key. Its request logs let you inspect individual requests and provider attempts.

Exported gateway traces include gen_ai.response.time_to_first_chunk alongside model and provider attributes. Group by both to identify combinations that slowed down. Compare requests with similar prompt lengths, output sizes, and settings before attributing the change to a provider.

Copy link to headingCost by model and feature

Per-model totals do not show which product feature generated the bill when several features share a model. Tag requests with a feature or environment, or attach a user identifier, to break down spending by those categories.

Set tags in providerOptions.gateway, alongside the routing settings.

1
import { generateText } from 'ai';
2

3
const ticket = `A customer upgraded their plan yesterday, but their dashboard
4
still shows the previous usage limit. Signing out and back in did not help.`;
5

6
const { text } = await generateText({
7
  model: 'anthropic/claude-fable-5',
8
  prompt: `Summarize this support ticket in two sentences:\n\n${ticket}`,
9
  providerOptions: {
10
    gateway: {
11
      models: ['anthropic/claude-opus-5', 'google/gemini-3.1-pro-preview'],
12
      tags: ['feature:ticket-summary', 'env:production'],
13
      user: 'user_123',
14
    },
15
  },
16
});
17

18
console.log(text);

The example prints a summary of the ticket and records its feature, environment, and user for reporting.

The Custom Reporting API groups spend and token usage by model, provider, tag, user, credential_type, and api_key_name. The AI SDK's gateway.getSpendReport() queries the same data. Each unique tag or user ID within a request counts as a metered write. Limit tags and identifiers to those you need for reporting.

Copy link to headingRetries and successful failovers

A request that fails on one provider and succeeds on another can appear as a success in the application's error-rate chart. Track the number of provider attempts per request and each attempt's outcome to identify retries hidden by the final response status.

AI Gateway Trace Drains export a trace for each sampled request, including provider-attempt spans. The attributes vercel.ai_gateway.attempt.success, error.type, and vercel.ai_gateway.attempt.provider_timeout record whether an attempt succeeded, its error, and whether it timed out.

Define each rate before building a dashboard:

• Multi-attempt request rate is the number of distinct requests with vercel.ai_gateway.attempt.total_in_request > 1, divided by all requests in the same period.

• Error-driven retry rate is the number of requests with a failed provider attempt followed by another attempt, divided by all requests in the same period.

• A provider's initial-attempt failure rate is the share of requests initially routed to that provider whose first provider attempt failed.

• Successful failover rate counts requests that succeeded on a subsequent provider or fallback model after an earlier failure. Specify whether the denominator is all requests or requests that needed another attempt.

Count distinct requests because a request with several attempts appears in several spans. Use the same request population and sampling policy for the numerator and denominator. To attribute failures to the initial provider, find the first provider attempt in the request. An attempt number can restart for a fallback model.

Multiple attempts do not always indicate a provider failure. AI Gateway's confidence-based decision fallbacks can run after a successful answer meets an escalation condition. Track those decision escalations separately from error-driven retries. The text-generation example above uses ordinary error fallbacks.

Zo Computer reported that its retry rate fell from 7.5% to 0.34% after moving calls to AI Gateway. Results will depend on the application's workload and providers.

Use model fallbacks and provider options to configure routing behavior, then use request logs to verify what happened.

Copy link to headingBudget consumption and alerts

AI Gateway budgets control spending for teams, projects, API keys, and team members. AI Gateway checks every applicable budget based on how the request is authenticated and which budgets its usage counts toward.

Budgets are soft caps. AI Gateway checks spending when a request starts, so the request that crosses a limit can complete. It rejects subsequent requests until the budget resets or its limit changes.

Budget rejections return HTTP 402 with the error type quota_for_entity_exceeded. AI SDK 7 can report that rejection as GatewayInternalServerError. Inspect the quota details before treating it as a provider incident or retrying.

Custom budgets can send optional alerts at 50%, 75%, and 100% of their limits. Alerts are off by default, and default budgets do not send them. Enable lower thresholds when you need a warning before rejection begins.

Copy link to headingThe seven tools in detail

Copy link to heading1. Vercel AI Gateway and AI SDK

Vercel AI Gateway provides one endpoint for hundreds of models and records usage, cost, latency, and provider attempts for routed requests. AI SDK telemetry adds application-level spans around supported model calls and tool execution.

AI Gateway traces show routing attempts. Application traces show the surrounding agent or request workflow, including work outside the gateway.

For application tracing, install ai, @ai-sdk/otel, and @vercel/otel. In a Next.js project, register the OpenTelemetry provider and AI SDK integration in instrumentation.ts:

import { registerOTel } from '@vercel/otel';
import { registerTelemetry } from 'ai';
import { OpenTelemetry } from '@ai-sdk/otel';

export function register() {
  registerOTel({
    serviceName: 'my-ai-app',
  });

  registerTelemetry(new OpenTelemetry());
}

Configure the OpenTelemetry provider to export to your chosen backend. After registration, supported SDK operations emit spans by default. Set telemetry.recordInputs and telemetry.recordOutputs to false to exclude a call's content.

The recommended OpenTelemetry integration uses gen_ai.* attributes. The SDK also provides LegacyOpenTelemetry for integrations that expect older ai.* spans. See the AI SDK telemetry documentation for recording settings and supported operations.

Advantages:

• Gateway metrics require no application tracing code beyond routing requests through AI Gateway.

• OpenTelemetry export lets teams send application traces to a compatible backend.

Limitations:

• Gateway delivery metrics do not establish answer correctness. Use evaluations to assess response quality.

• Calls sent directly to a provider bypass gateway metrics. Work outside supported SDK operations needs its own instrumentation.

The dashboard and logs are included with AI Gateway. Extended timeframes and retention require Observability Plus. Custom Reporting costs $0.075 per 1,000 tag or user ID writes and $5 per 1,000 queries. Trace Drains cost $0.05 per 1,000 delivered traces plus $0.50 per GB of egress. Reporting and gateway trace export require eligible Pro or Enterprise plans; inference has separate charges. For the latest prices, see AI Gateway pricing.

Copy link to heading2. Langfuse

Langfuse is an LLM engineering platform for tracing, evaluation, prompt management, and datasets. Its core is MIT-licensed, and the company is part of ClickHouse. It models trace steps as observations, such as generations, tools, retrievers, and evaluators.

The AI SDK 7 integration uses @langfuse/vercel-ai-sdk with LangfuseSpanProcessor from @langfuse/otel. In serverless deployments, complete trace flushing before the function terminates. On Vercel, supported post-response work can use after() or waitUntil(), depending on the framework.

Teams can use Langfuse Cloud or self-host it. Before moving between them, check which enterprise features require a license and what infrastructure the deployment needs.

Advantages:

• Teams can run the MIT-licensed core on their own infrastructure and control where traces are stored.

• Prompt versions link to observations, so teams can inspect the traces produced by a particular prompt.

Limitations:

• Self-hosting requires managing storage, upgrades, backups, and scaling. It does not control how external model or evaluation services store data.

• The vendor integration adds Langfuse-specific configuration. A later migration may require changing that configuration or attribute mapping.

Langfuse Cloud pricing includes a free Hobby plan with 50,000 units per month and 30 days of data access. Core costs $29 per month with 100,000 units and 90 days of access. Pro costs $199 per month with three years of access. The self-hosted core has no software license fee; infrastructure and operations still cost money.

Copy link to heading3. LangSmith

LangSmith provides observability and evaluation for AI applications, including LangChain and LangGraph workflows. It supports annotation queues, prompt management, production monitoring, and LangSmith Engine, which groups failures into issues and proposes fixes.

AI SDK applications can use LangSmith without LangChain. For AI SDK 7 with langsmith 0.7.2 or later, import LangSmithTelemetry from langsmith/experimental/vercel and register it with registerTelemetry(LangSmithTelemetry()).

LangSmith also accepts OTLP traces. Its separate LLM Gateway handles model fallbacks and records gateway events in traces. You can use the tracing integration without routing requests through that gateway.

Advantages:

• Compatible LangChain and LangGraph applications can enable built-in tracing through environment configuration without wrapping individual calls.

• LangSmith Engine groups related failures and can propose prompt or code changes for review.

Limitations:

• The proprietary backend requires an Enterprise agreement for self-hosting, which may conflict with an open-source requirement.

• The AI SDK integration uses an experimental import path, so check compatibility when upgrading.

The Developer plan includes 1 seat and 5,000 base traces per month at no subscription cost, with charges for usage beyond the allowance. Plus costs $39 per seat per month and includes 10,000 base traces, followed by usage-based charges. Gateway and other services have their own usage terms.

Copy link to heading4. Braintrust

Braintrust lets teams save production traces as evaluation test cases. They can score those cases with custom evaluators and compare prompt or model changes before release.

For AI SDK 7, initialize Braintrust with initLogger() and the project and API-key configuration, then register braintrustAISDKTelemetry() before making AI SDK calls. Braintrust's automatic instrumentation is another option. Braintrust also supports OpenTelemetry export and accepts gateway traces through the Vercel Marketplace integration.

Braintrust maps GenAI and OpenInference attributes into its trace fields. It also supports integrations that use older ai.* attributes.

Advantages:

• Teams can turn production traces into evaluation datasets without exporting and reimporting them.

• The Loop agent can propose changes to prompts, scorers, and datasets that teams can validate through experiments.

Limitations:

• On-premises deployment is part of the custom-priced Enterprise offering.

• Braintrust bills processed data, scores, and model usage separately. Include them when estimating costs.

The Starter plan has no monthly platform fee and includes 1 GB of processed data, 10,000 scores per month, and 14-day retention. Additional processed data costs $4 per GB, and additional scores cost $2.50 per 1,000. Pro costs $249 per month with 5 GB, 50,000 scores, and 30 days of included retention, followed by usage charges.

Copy link to heading5. Arize Phoenix and Arize AX

Arize Phoenix is a source-available platform for local tracing, evaluation, experimentation, and prompt iteration. Arize AX is a separate managed platform with its own plans and pricing.

The OpenInference integration uses @arizeai/openinference-vercel to convert compatible AI SDK spans into OpenInference attributes. For AI SDK 7, the Arize integration uses LegacyOpenTelemetry before conversion. Attributes include llm.token_count.total, with other prefixes for tools and retrieval.

The OpenTelemetry GenAI conventions remain in development. Check convention versions and backend attribute support when choosing either format.

Advantages:

• Phoenix runs locally, so developers can inspect model calls and tool execution before choosing a managed backend.

• Developers can replay traced calls in the playground and add them to evaluation datasets.

Limitations:

• Phoenix's AI views recognize more fields in OpenInference spans than in other formats. Check which fields appear when sending those traces to another backend.

• Phoenix uses Elastic License 2.0, or ELv2. It restricts offering substantial software functionality to third parties as a hosted or managed service. The Phoenix license distinguishes that use from internal self-hosting.

Phoenix has no software license fee for permitted self-hosted use. Teams remain responsible for infrastructure and operations. Separately, AX Free includes 25,000 spans and 1 GB of ingestion per month, with 15-day retention. AX Pro costs $50 per month with 50,000 spans, 10 GB of ingestion, and 30-day retention.

Copy link to heading6. Datadog Agent Observability

Datadog Agent Observability traces LLM and agent workflows alongside application monitoring. It records tokens and costs, supports evaluations and security analysis, and connects agent traces with backend service traces.

Datadog supports OpenTelemetry GenAI semantic conventions. AI SDK applications can export compatible spans through an OpenTelemetry provider, with intake credentials such as DD_API_KEY configured on the server. The Datadog integration for Vercel also connects function logs, traces, and metrics.

When teams instrument the relevant services, they can trace a slow agent response to a retrieval step or database query in Datadog.

Advantages:

• Teams can investigate model calls, tools, and backend services on a single monitoring platform.

• Datadog accepts compatible GenAI spans without a vendor-specific AI SDK telemetry integration.

Limitations:

• Connecting agent and infrastructure traces requires APM and service instrumentation.

• Costs increase with model-call volume. Datadog's Agent Observability meter bills LLM spans, excluding tool, workflow, and retrieval spans.

The free tier includes 40,000 LLM spans per month with 15-day retention. Pro includes 100,000 LLM spans and starts at $160 per month with an annual commitment, or $200 per month on a month-to-month basis. Additional usage and other Datadog products have separate charges.

Copy link to heading7. Sentry

Sentry combines error monitoring and performance tracing with agent views for AI applications. It records model and tool calls, token usage, costs, and errors, along with application exceptions and releases.

Sentry's AI SDK 7 integration uses the native Node.js telemetry channel. It requires a Sentry server SDK such as @sentry/node or @sentry/nextjs version 10.62.0 or later. Initialize Sentry and enable tracing; no separate @ai-sdk/otel package or registerTelemetry call is required for this integration.

Sentry is also a native destination for AI Gateway Trace Drains. Configure both sources when you need application steps and gateway routing attempts in Sentry.

Advantages:

• Teams can investigate agent failures alongside stack traces, releases, and application errors.

• Node.js applications with Sentry tracing configured can capture supported AI SDK calls through the telemetry channel.

Limitations:

• This AI SDK integration requires the Node.js runtime.

• Dedicated LLM-as-judge evaluation workflows need a separate evaluation tool.

If an application also uses a custom OpenTelemetry provider, follow the combined setup in Vercel's tracing documentation to avoid conflicts with Sentry's provider.

The Developer plan is free for one user and includes 5,000 errors and 5 million spans per month. Team starts at $26 per month and Business at $80 per month when billed annually with the default prepaid data allowance. Additional usage can change the total.

Copy link to headingHow to choose an AI observability tool

Choose based on the failures you need to investigate and the tools your team already uses:

• For model cost, latency, and provider failures, Vercel AI Gateway records metrics and routing attempts for requests sent through it. Add application tracing to inspect the surrounding work.

• For evaluations and prompt management with a self-hosting option, compare Langfuse with Phoenix. Check their operating requirements and license terms.

• For LangChain or LangGraph applications, LangSmith's built-in tracing can reduce integration work. Its AI SDK integration also works without those frameworks.

• To build regression tests from production traces, assess how Braintrust's datasets and experiments align with your evaluation process.

• To investigate model calls alongside infrastructure already monitored by Datadog APM, use Datadog Agent Observability.

• If your team already uses Sentry, its agent views let you investigate agent failures alongside application exceptions and release regressions.

Check vendor support, export options, and integration maintenance before adopting a tool.

Test portability by exporting a run with model calls, tool execution, and errors to a second backend. Check that it preserves the fields and relationships your team needs, including session identifiers and evaluation scores.

To configure an agent, follow Build AI agents with AI Gateway and AI SDK. If you start from a template, check its gateway and telemetry configuration. Instrument retrieval, database calls, or business logic that the SDK does not trace.

Copy link to headingFrequently asked questions

Copy link to headingWhich AI observability tools have free plans?

AI Gateway includes its dashboard and logs with routed traffic. Inference and optional add-ons are charged separately. Langfuse's MIT-licensed core and Phoenix can be self-hosted without software license fees. Langfuse Cloud, LangSmith, Braintrust, Arize AX, Datadog, and Sentry have free allowances. Compare what each plan meters, how long it retains data, and what happens when usage exceeds the allowance.

Copy link to headingIs AI Gateway enough for AI observability?

AI Gateway records cost, latency, usage, and provider attempts for requests routed through it. Application tracing is needed to investigate retrieval, tool execution, and other work outside the gateway. Evaluations assess response quality and help detect regressions.

Copy link to headingHow does gateway observability differ from agent observability?

Gateway observability follows a model request through routing and provider attempts. Agent observability follows a run through model calls, tools, and intermediate steps. Sessions can group several runs. Both sources can send traces to the same backend. Check trace propagation and correlation before treating them as one connected trace.

Copy link to headingAre OpenTelemetry GenAI conventions stable?

The GenAI conventions remain in development, so attributes can change between versions. The AI SDK's OpenTelemetry integration emits GenAI attributes while LegacyOpenTelemetry retains older ai.* attributes for integrations that expect them. Pin compatible versions and verify backend field mappings when upgrading.