Model retirements now force a migration every year or so, and every prompt change is a release. A small, honest eval set turns both into a day of work instead of a month of guessing. How to build one from real traffic, how to grade it, and what to automate.
Bedrock retired Claude Sonnet 4 in October 2026 and has more retirements scheduled through 2027. Every team on those models has to move, and the ones with an eval set will do it in a day: run the set on the new model, read the diffs, adjust two prompts, ship. The ones without will move and wait for complaints.
Real traffic first. Sample production inputs, stratified by type, and include the long, messy and ambiguous ones.
Every bug report becomes a case. The cases that once failed are the most valuable ones you have.
Edge cases you know about: empty input, other languages, adversarial phrasing, inputs that should be refused.
Start with 50 to 200 cases. A small set someone reads beats a large set nobody trusts.
Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Run the set on every prompt or model change, in CI, and fail the build on regressions in the programmatic checks.
Report judge scores as a diff against the last release, case by case, so a reviewer reads the ten that changed rather than a single average.
Track cost and latency per case alongside quality. A model that is 2% better and 3x the price is a business decision, not an engineering one.
Keep the set versioned with the prompts it tests.
Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.
Book a free 30-minute call about it. Bring your setup and your bill; you'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Or keep reading
pgvector on RDS, Measured: HNSW and IVFFlat Build Times and Latency on 50,000 Vectors
On 50,000 vectors of 768 dimensions on a 2-vCPU RDS instance, an exact nearest-neighbour search took 133 ms. HNSW cut it to 1.9 ms after a 47-second build; IVFFlat to 1.8 ms after 2.4 seconds. What recall looked like, why our numbers are a floor, and how to size this for real embeddings.
Read
What MCP Is: Hosts, Clients, Servers, Tools, and How Sign-In Works
MCP (the Model Context Protocol) is the standard way an AI application connects to tools and data: Claude, ChatGPT, Cursor and VS Code speak it, and any company can publish a server. MCP explained in plain terms: the parts, what a server exposes, the two transports, how authorization works since the 2026 spec, where the security risks are, and when building a server is worth it.
Read
As of October 25, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.