Updated October 24, 2026
· ~3 min read
An LLM gateway sits between your applications and the model providers. It gives every team one API, holds the provider keys so applications don't have to, and enforces the things finance and security will eventually ask for: budgets per team, rate limits, fallbacks when a provider is down, caching, and an audit trail of who sent what.
It is also, by design, the most sensitive box in your AI stack — it holds every key and sees every prompt. 2026 made that concrete. LiteLLM, the most widely used open-source gateway, had malicious releases published twice: in March through stolen PyPI credentials, and on October 21 through its own compromised release pipeline, with a credential stealer shipped to PyPI and the official Docker images. Portkey, meanwhile, was acquired by Palo Alto Networks and is becoming part of its Prisma AIRS security platform.
If you are already on AWS, Amazon Bedrock deserves a place in this comparison too: for models it hosts, it already provides one API, IAM-based access, guardrails and logging, with no gateway to run.
LiteLLM is still a capable gateway, and both compromises reached only teams that installed unpinned versions or followed floating tags. If you run it: pin an exact, verified version and image digest; pull through CodeArtifact or an ECR pull-through cache so new upstream releases need an explicit decision; keep provider keys in Secrets Manager and give the task its own narrow IAM role; restrict its egress to the providers it calls; and alert on unexpected outbound connections. If you installed litellm 1.107.0 or pulled the affected tags on October 21, follow LiteLLM's incident guidance and rotate every credential the environment could reach.
Question 1 of 3
Are the models you need available on Amazon Bedrock?
If the models you need are on Amazon Bedrock, start there: one Converse API across providers, IAM instead of shared API keys, Guardrails, CloudWatch and CloudTrail logging, and nothing extra to operate or patch. Add a gateway only when you need providers Bedrock doesn't host or cross-provider routing and budgets. For that, LiteLLM remains the most capable open-source option — run it pinned to a verified version, pulled through your own registry, with tight egress. Choose Portkey if you want a managed gateway and its direction inside Palo Alto Networks suits you; Kong AI Gateway if you already run Kong; and OpenRouter for prototypes and products that want the widest model catalogue with zero infrastructure — it is a managed third party, so check its data terms before sending it anything sensitive.
Secure LLM Gateway on AWS
One governed way for every team to call AI models: keys held centrally, budgets and rate limits per team, fallbacks between providers, and hardening that assumes the gateway is a target.
LLM Observability & Evaluation on AWS
See what your AI features actually do: every prompt, response, cost and latency traced, and evaluations that tell you whether a change made things better or worse. Running in your own AWS account.
Amazon Bedrock Enterprise AI Platform
A production AI platform on Amazon Bedrock: RAG pipelines, agent orchestration, guardrails, multi-tenant cost tracking, and the observability needed to run AI in production.
We work with startups and growing companies on AWS architecture decisions. Tell us about your situation and we will recommend the right approach.
As of October 24, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.