With a few thousand visitors a month, most A/B tests can only detect enormous differences, and most "winners" are noise. The arithmetic, what to test instead of button colours, how to count exposures properly, and the rules we use on our own site.
To detect a change from a 3% conversion rate to 4.5%, a 50% improvement, with the usual 95% confidence and 80% power, you need about 2,500 visitors in each variant. To detect 1% becoming 1.5%, about 7,700 each. A site with 3,000 visitors a month testing a 1% conversion would wait five months for one answer, and a 10% improvement would never be detectable at all.
This is why most small-site tests "find" winners: they stop when the chart looks good, and with small numbers the chart looks good by chance a lot of the time.
Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
The offer: a free assessment against a fixed-price package.
The positioning: a senior individual against a team — the test we are running on our own hero line.
Which service leads the home page.
Whether pricing is shown.
Differences this large can move conversion by tens of percent, which a small site can detect in weeks. Button colours and microcopy cannot.
A visitor who never scrolled to the variant did not see it. Counting them dilutes both arms equally and hides a real difference, so record an exposure only when the variant is actually on screen — we record it once per page view when half the slot is visible. Assign by a hash of the visitor id so the same person always sees the same variant, and analyse per visitor, not per page view, so one enthusiastic reader cannot swing the result.
Decide the sample size and the success event before starting, and do not look at significance until you reach it.
Report a range, not a single number: if the two variants' 95% ranges overlap, the honest answer is "not yet".
Use a leading metric when the real goal is rare — clicks toward booking alongside bookings — but decide on the real goal.
One test per page at a time.
When a test ends inconclusive, keep the variant you prefer for other reasons. "No detectable difference" is a result.
Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.
Book a free 30-minute call about it. Bring your setup and your bill; you'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Or keep reading
Ad-Hoc Questions on DynamoDB vs Postgres: 16 Analytics Queries at 100,000, 1 Million and 5 Million Orders
Every question across customers is a full table scan in DynamoDB: 217,000 read units at 5 million orders, whether it returns 10 rows or all of them. Postgres answered the same 16 questions in 0.08 to 2.7 seconds without a single index added for them, and an index built in 2.7 seconds took one from 148 ms to 44 ms. Measured in AWS.
Read
What a DynamoDB Scan Costs: The Dashboard That Refreshes Every Five Minutes
A Scan of a million 327-byte orders reads about 40,000 request units, half a cent. Run it every five minutes for a dashboard and it is $43 a month; at 100 million orders, $4,337. The arithmetic from measured item sizes, and the four alternatives: Stream-fed aggregates, exports to S3 with Athena, zero-ETL to an analytics store, or Postgres.
Read
As of October 26, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.