Secure LLM Gateway on AWS

One governed way for every team to call AI models: keys held centrally, budgets and rate limits per team, fallbacks between providers, and hardening that assumes the gateway is a target.

Overview

An LLM gateway holds every provider key and sees every prompt, which makes it the most sensitive component in your AI stack. In 2026 the most popular open-source gateway, LiteLLM, shipped malicious releases twice — once through its own release pipeline. We set up the gateway your teams need and harden it as the target it is.

Who this is for

• Companies where several teams call AI models with their own keys and nobody can see the total

• Teams needing models Bedrock doesn't host, or routing and fallback across providers

• Anyone running LiteLLM who wants to know they weren't exposed — and won't be next time

What we implement

The right gateway for you

• Amazon Bedrock directly where the models you need are there: one Converse API, IAM auth, Guardrails, CloudTrail logging

• A self-hosted gateway — LiteLLM or Kong AI Gateway — where you need other providers or cross-provider routing

• Portkey or OpenRouter where a managed service is the better trade, with their data terms reviewed

Hardening

• Exact versions and image digests pinned, pulled through CodeArtifact and an ECR pull-through cache so upstream releases need an explicit decision

• Provider keys in Secrets Manager, a dedicated least-privilege IAM role, and egress restricted to the providers you use

• Alerts on unexpected outbound connections and on spend anomalies

Governance

• Virtual keys per team or application, with budgets, rate limits and model allow-lists

• Fallbacks and retries across providers and regions

• Request logs and cost attribution, feeding your observability stack

Deliverables

• A running, hardened gateway (or a Bedrock-native setup) with IaC

• Per-team keys, budgets and dashboards

• A runbook for upgrades and for responding to a compromised dependency