FINRA processes about 37 billion market records on an average day and more than 75 billion on busy ones. It moved surveillance to AWS because fixed capacity was getting in the way of analysis, put S3 at the center, and separated storage from compute. The architecture, the outcomes and what it means for smaller data platforms.
FINRA, which regulates US broker-dealers, moved its market surveillance analytics to AWS because, in its words, fixed capacity and provisioning lead times were getting in the way of the analytics.
S3 became the single store for market data, with compute (EMR, and HBase for some workloads) scaled independently against it. An open-source catalog tool, herd, tracked datasets and lineage.
On an average day the platform processes about 6 TB and 37 billion records, and more than 75 billion records on busy days, with more than 600 TB archived and 300 million objects in S3 at the time AWS described it.
Analysts could query years of history in minutes or hours instead of weeks or months, run many workloads against the same data at once, and use Spot capacity to lower cost.
The Consolidated Audit Trail, built later for the whole US market on AWS, ingests more than 100 billion market events a day from 22 exchanges and around 1,500 broker-dealers.
Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Separating storage from compute is the decision that aged best. It is now the default design for warehouses and lakehouses (Iceberg on S3, queried by Athena, Redshift, EMR or Snowflake) at every scale.
FINRA's phrase about lowering "the cost of curiosity" is the real benefit. When a question costs a query instead of a capacity request, people ask more questions.
You do not need FINRA's volume to use the pattern. The same shape runs a startup's event analytics for a few dollars a month.
Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.
Book a free 30-minute call about it. Bring your setup and your bill; you'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Or keep reading
How Capital One Left Eight Data Centers for AWS, and What the 2019 Breach Taught Everyone
Capital One was the first US bank to announce going all in on the cloud, closing eight data centers by 2020 and rebuilding 80% of about 2,000 applications as cloud native. What changed in delivery speed and reliability, and the security lesson from 2019 that changed EC2 for every customer.
Read
What Ten Enterprise Modernizations Have in Common: Home Depot, Lowe's, Capital One, Netflix, Amazon and More
Ten public modernization stories, side by side: where each started, how long it took, how much was rebuilt rather than moved, and what changed. Six patterns repeat in almost all of them, and each one applies to a company far smaller than these.
Read
As of October 25, 2026. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.