A data lakehouse combines the scale of a data lake with the query performance and governance of a warehouse. The AWS services we use to run one in production, and how they fit together.
A data lakehouse is an architecture that combines the storage scale and flexibility of a data lake with the ACID guarantees, schema enforcement, and query performance of a data warehouse. In practice on AWS, this means S3 as the storage layer, Apache Iceberg as the table format, AWS Glue as the catalog and ETL engine, and Athena or Redshift as the query layer.
The key insight: you do not need Redshift or Databricks to get warehouse-quality query performance on your S3 data. Apache Iceberg, running on Athena or Spark, gives you ACID transactions, time travel, and schema evolution directly on your S3 data lake. This eliminates the need to ETL data into a separate warehouse for many use cases.
S3: the storage foundation — cheap, durable, and the source of truth for all data. Everything else is a processing or query layer on top of S3.
Apache Iceberg: the table format that turns S3 files into a queryable, transactional table. Iceberg metadata tracks file manifests, snapshots, and schema history.
AWS Glue Data Catalog: the metadata layer — stores table schemas, partition information, and database definitions. Acts as the Hive Metastore for Spark, Athena, and Redshift Spectrum.
AWS Glue ETL: managed Spark and Python jobs that transform raw data into processed Iceberg tables. Integrated with the Glue Catalog for schema discovery.
Amazon Athena: serverless SQL query engine on S3. Full Iceberg support including MERGE and time travel. Pay per query based on data scanned.
Amazon Redshift: columnar warehouse for heavy BI workloads, concurrent dashboard queries, and complex joins on large datasets. Can query S3 via Redshift Spectrum.
AWS Lake Formation: governance layer on top of Glue Catalog — column-level security, fine-grained access control, and cross-account data sharing.
Traditional S3 data lakes use Hive-style partitioning: files are organized in directories by partition values (year=2024/month=03/day=15/). This works but has major limitations: no atomic writes (readers can see partial data), no row-level deletes (you must rewrite entire partitions), no schema evolution without breaking downstream queries, and no time travel.
Apache Iceberg solves all of these. Iceberg maintains a tree of metadata: the current snapshot points to a manifest list, which points to manifests, which point to the actual Parquet or ORC data files. Writes are atomic — the metadata pointer is updated only after all data files are written. Row-level deletes use delete files that are compacted during maintenance. Schema changes (adding columns, renaming, changing types where compatible) are metadata-only operations. Time travel queries any historical snapshot.
Book a free 30-minute call. You'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Raw zone (bronze): exact copy of source data, immutable, never modified. S3 Object Lock prevents deletion. This is your source of truth for reprocessing.
Processed zone (silver): cleaned, typed, partitioned Iceberg tables. Schema enforced. Nulls handled. PII tokenized. This is what analysts query for operational work.
Curated zone (gold): business-logic aggregates, pre-joined tables, and denormalized views optimized for specific reporting use cases.
Archive zone: data past retention policy, moved to S3 Glacier via lifecycle rules. Typically 3–7 years for compliance.
Athena and Redshift are not competing — they are complementary. Athena is the right choice for ad-hoc exploration, operational queries, and low-frequency reporting. It scales to zero when not used, costs nothing in idle, and supports full Iceberg (including MERGE and time travel). Redshift is the right choice for high-concurrency BI dashboards, complex joins on multi-terabyte datasets, and workloads where query latency must be consistently sub-second. Redshift's columnar storage and result caching make it significantly faster for repeated queries at scale.
The common pattern: use Athena for everything while your data team is small, switch workloads to Redshift as dashboard concurrency grows and p99 query latency becomes a problem. Redshift Spectrum lets Redshift query your S3 Iceberg tables directly, so you can serve some queries from Redshift without moving data out of the lake.
Lake Formation adds a permission model on top of the Glue Data Catalog — column-level access control, table-level sharing across accounts, and row filters. The overhead is real: every query goes through a Lake Formation permission check, IAM policies must grant both S3 access and Lake Formation permissions, and the configuration complexity increases substantially. Use Lake Formation when: you need column-level security (e.g., masking PII for non-privileged analysts), you need to share specific tables to external accounts without sharing S3 buckets, or you are subject to regulatory requirements that mandate fine-grained access controls. For small teams in a single account without compliance requirements, plain S3 bucket policies and Glue Catalog permissions are sufficient.
S3 storage: cheap — $0.023/GB-month for Standard, $0.0125/GB-month for Infrequent Access. Optimize with lifecycle policies to move cold data to IA or Glacier.
Athena: $5/TB scanned. Optimize aggressively with Parquet/ORC format (10–20× cheaper than scanning CSV), partition pruning (queries that filter on partition columns scan only relevant files), and CTAS for repeated heavy queries.
Glue DPU-hours: $0.44/DPU-hour. A 10-DPU nightly job running 2 hours costs ~$8.80/night or $264/month. Reduce with smaller G.025X workers for simple jobs and time-windowed scheduling.
Redshift: the biggest line item if you run it. An ra3.xlplus cluster (2 nodes) costs ~$1,400/month. Redshift Serverless charges $0.36/RPU-hour and scales to zero — better for unpredictable workloads.
Once a month: what changed in tech that month and what it means for you, from cloud and AI to security and the tools you build with. Every item links to its source. No sales pitch, unsubscribe in one click.
Book a free 30-minute call about it. Bring your setup and your bill; you'll leave knowing what we'd do first and roughly what it costs, whether or not you hire us.
Or keep reading
AWS Glue vs EMR vs Athena: Choosing the Right Data Processing Tool
Glue, EMR and Athena overlap on the surface. What each is good at, what each costs, and how to choose.
Read
First-Party Analytics on AWS for a Few Dollars a Month: The Pipeline Behind This Site
A visitor id, a session id and a page view id; a collector on our own domain; Firehose into S3; Athena with saved queries. How we built analytics that ad blockers do not drop, nobody else holds, and that costs less than a coffee at our traffic, and the three bugs we found on the way.
Read
As of February 10, 2025. Provided as is, for general information, and accurate to the best of our knowledge on the date shown. Not legal, compliance, tax, financial or other professional advice. Prices, limits and vendor services change; check current vendor documentation before relying on anything here. Terms of use.