Updated October 24, 2026
· ~3 min read
Once an LLM feature is in front of customers, three questions arrive at once: what did the model actually see and say (tracing), is it getting better or worse (evaluation), and what is it costing (token and latency accounting). LLM observability platforms answer all three, and the market for them consolidated hard in 2026.
ClickHouse acquired Langfuse in January and kept its core MIT-licensed and self-hostable. Mintlify acquired Helicone in March and put it into maintenance mode — it still runs and gets security fixes, but new sign-ups are closed and the roadmap is frozen. Braintrust raised a large round and is pushing from evaluation into observability. LangSmith remains the native choice for LangChain and LangGraph shops, and Arize Phoenix is the source-available, OpenTelemetry-first option for teams that lead with evaluation.
This guide compares them on what decides a production choice: data ownership, self-hosting on AWS, OpenTelemetry support, evaluation depth, and how they fit Amazon Bedrock and AgentCore.
An observability platform ends up holding every prompt, every completion and often personal data your users typed. That makes "who owns this next year, and can I run it myself" a real requirement rather than a nice-to-have. Helicone is the cautionary example: a well-liked product that stopped taking new customers overnight. Langfuse's acquisition went the other way — the MIT core and self-hosting were kept, though supported self-hosted enterprise now comes bundled with a ClickHouse commercial plan. If the data matters, prefer a tool you can run in your own AWS account under a license you are comfortable with, and keep your instrumentation on OpenTelemetry so changing tools later is a configuration change rather than a rewrite.
A production Langfuse in your own account is a web and a worker container (ECS on Fargate is the simplest home), Amazon RDS for PostgreSQL, ClickHouse for the trace store (self-managed on EC2, or ClickHouse Cloud in your region), Redis or Valkey on ElastiCache for the queue, and S3 for event and media storage. Put it behind an ALB with SSO in front, keep it in a private subnet, and size ClickHouse for your trace volume — that is where the cost is. For teams that don't want to run it, Langfuse Cloud is the same product, managed.
Question 1 of 4
Must prompts and completions stay inside your own AWS account?
For most teams on AWS, Langfuse is the default: MIT-licensed, self-hostable in your own account (or used as a managed cloud), OpenTelemetry-native, framework-agnostic, and with first-class Amazon Bedrock AgentCore support. Choose LangSmith if your application is built on LangChain or LangGraph and you want the tightest integration with them. Choose Arize Phoenix if evaluation and experimentation come first and you want an OpenTelemetry-native tool you can run anywhere — reading its Elastic License 2.0 terms first. Choose Braintrust if systematic evaluation is the core of how you ship and a managed product suits you. Do not start anything new on Helicone; plan a migration if you already depend on it.
LLM Observability & Evaluation on AWS
See what your AI features actually do: every prompt, response, cost and latency traced, and evaluations that tell you whether a change made things better or worse. Running in your own AWS account.
Secure LLM Gateway on AWS
One governed way for every team to call AI models: keys held centrally, budgets and rate limits per team, fallbacks between providers, and hardening that assumes the gateway is a target.
Production GenAI Platform on AWS
Deploy secure, scalable AI systems using AWS-native GenAI services. RAG, agent orchestration, and guardrails: production-ready.
We work with startups and growing companies on AWS architecture decisions. Tell us about your situation and we will recommend the right approach.
As of October 24, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.