A production data lakehouse on AWS (S3-backed data lake with Glue, Iceberg, Athena, and Redshift) structured for analytics, ML feature stores, and governed data sharing.
4–10 weeks
$25k–$60k
A data lake stores raw data in its native format on cheap object storage (S3). Flexible and cheap, but lacks ACID guarantees, schema enforcement, and query performance. A data warehouse (Redshift, BigQuery) provides ACID, schema enforcement, and fast SQL queries, but requires ETL to load data in and is expensive to store large volumes. A data lakehouse combines both: S3 as the storage layer (cheap, durable, scalable), with Apache Iceberg as the table format that adds ACID transactions, schema enforcement, and time travel on top.
S3 is the source of truth for all data in the lakehouse. We organize data into zones:
Raw zone (bronze): exact copies of source data — database exports, API responses, log files. Immutable. S3 Object Lock prevents modification or deletion. This is your reprocessing safety net.
Processed zone (silver): cleaned, typed, partitioned Apache Iceberg tables. Nulls handled, PII tokenized or removed, data types standardized. This is the primary analytical layer.
Curated zone (gold): business-logic aggregates, pre-joined denormalized tables, and reporting-optimized views. These tables are optimized for specific BI use cases.
Archive zone: data past retention policy, moved via S3 lifecycle rules to S3 Glacier Instant Retrieval (for compliance data that may need retrieval) or Deep Archive (for cold backup).
S3 lifecycle policies move data automatically: Standard → Intelligent-Tiering → Standard-IA → Glacier, typically based on last-access date. On large datasets this reduces S3 storage costs by 40–70%.
Apache Iceberg transforms S3 files into a queryable, transactional table format. The key capabilities:
ACID transactions: atomic writes using optimistic concurrency control. Readers always see a consistent snapshot. Multi-statement transactions are supported via Spark.
Time travel: query any historical snapshot — `SELECT * FROM table FOR SYSTEM_TIME AS OF TIMESTAMP '2024-01-01'`. Critical for debugging, auditing, and reprocessing.
Schema evolution: add, rename, reorder, or widen columns without rewriting data files. Type promotion (int → long, float → double) is metadata-only.
Partition evolution: change your partition scheme (e.g., from daily to hourly partitioning) without rewriting existing data. Old files maintain old partition structure; new files use the new scheme.
Row-level deletes: Iceberg delete files enable GDPR right-to-erasure compliance and CDC MERGE operations without rewriting entire partitions.
AWS Glue serves two distinct roles: the Glue Data Catalog (metadata store for all tables) and Glue ETL (managed Spark and Python jobs).
The Glue Data Catalog acts as the Hive Metastore for Spark (via EMR or Glue), Athena, and Redshift Spectrum. All three query engines use the same catalog — a table defined once in Glue is queryable from all three without duplication.
Glue ETL jobs run managed Apache Spark. Glue 4.0 (Spark 3.3) includes native Iceberg support — you can read and write Iceberg tables directly from Glue jobs without additional configuration. We use Glue for nightly batch ETL, CDC processing from DMS output, and data quality checks via Glue Data Quality (rule-based data validation with alerting).
Batch ingestion: AWS Database Migration Service (DMS) for full loads and CDC from RDS/Aurora/SQL Server/Oracle → S3. Glue jobs transform DMS output into Iceberg tables via MERGE operations.
Streaming ingestion: Kinesis Data Firehose buffers events (clickstream, IoT telemetry, application events) and delivers to S3 in Parquet format. Glue Streaming Jobs or Lambda processes the Firehose output into Iceberg.
Partner data drops: S3 event notifications trigger Lambda or Glue jobs when partner files (CSV, XML, JSON) arrive in designated S3 prefixes.
Athena is the right query engine for: ad-hoc SQL exploration by data analysts, operational reporting (hourly refresh), API-driven queries from applications, and full Iceberg MERGE/time-travel operations. Cost is $5/TB scanned — optimize with Parquet format, partition pruning, and query result reuse. Athena scales to zero when not in use.
Redshift is the right choice for: concurrent BI dashboard queries (10+ simultaneous users), heavy aggregations on multi-terabyte fact tables, complex multi-table joins in reporting queries, and workloads where sub-second p99 query latency is required. Redshift Serverless charges $0.36/RPU-hour and scales to zero — better than provisioned for unpredictable workloads.
Redshift Spectrum lets Redshift query your S3 Iceberg tables directly, without loading data into Redshift storage. This allows you to serve some queries from Redshift without duplicating your lake data.
Lake Formation adds a permission model on top of the Glue Catalog: column-level security (mask PII columns for non-privileged users), table-level sharing to external AWS accounts, and row-level filters. Use Lake Formation when column-level security is a requirement. For teams without strict column-level governance needs, S3 bucket policies and Glue Catalog permissions are simpler and sufficient.
We use AWS Step Functions for ETL orchestration in most environments: native AWS service, no additional infrastructure, excellent integration with Glue and Lambda, visual workflow editor, and built-in retry and error handling. For teams with existing Airflow expertise or complex dependency graphs, Amazon MWAA (Managed Workflows for Apache Airflow) is the alternative.
S3 bucket structure with lifecycle policies and zone definitions
Apache Iceberg table design for your primary datasets
Glue Data Catalog with all source and derived tables
Glue ETL jobs for batch ingestion and CDC processing
Athena workgroup configuration with cost controls
Lake Formation permissions (if column-level security required)
Orchestration pipeline (Step Functions or MWAA)
Data quality checks via Glue Data Quality
BI layer configuration (QuickSight or Redshift)
Runbooks: data pipeline operations, schema changes, cost optimization
Tell us about your current environment and goals. We'll review the details and respond with a proposed engagement structure and timeline.
Typical response time: 1–2 business days.
Prices and timelines are estimates for typical engagements, not offers. Scope, price and terms are set in a written agreement after we understand your situation. Terms of use.