Large-Scale Data Platform on Azure & Spark
Build a unified data platform processing 50TB daily from 200+ sources, replacing siloed data warehouses and enabling real-time analytics for a global retail operation.
Business Context
Multi-billion dollar retailer with operations in 30 countries. Existing architecture was a maze of point-to-point integrations, inconsistent data definitions, and 48-hour latency for business insights.
Scale & Constraints
50TB daily ingestion, 200+ source systems, 5000+ downstream consumers, strict data freshness SLAs (15 min for critical paths), cross-region replication requirements.
Architectural Role
Platform architect and technical lead. Defined reference architecture, established data contracts, led vendor evaluations, and governed implementation across 8 engineering teams.
System Design
Medallion architecture on Azure Data Lake: Bronze (raw ingestion via Kafka/Event Hubs) → Silver (cleansed, typed, deduplicated via Spark) → Gold (business aggregates, curated datasets). Unity Catalog for governance. Delta Lake for ACID transactions. Synapse for SQL analytics layer.
Key Architectural Decisions
- ✓Delta Lake over Parquet—ACID transactions justified 15% storage overhead
- ✓Medallion over Lambda—simplified architecture, accepted batch latency for most use cases
- ✓Centralized schema registry—forced contracts, enabled breaking change detection
- ✓Databricks over HDInsight—TCO analysis showed 30% savings at scale
Trade-offs
- ⚠Chose eventual consistency for cross-region—simplified architecture but required consumer awareness
- ⚠Batch processing for 80% of pipelines—real-time only where SLAs demanded it
- ⚠Single cloud vendor lock-in for data layer—accepted for operational simplicity
Technologies
Measurable Outcomes
- →Data freshness improved from 48 hours to 15 minutes
- →60% reduction in data engineering effort via self-service
- →$4M annual savings from warehouse consolidation
- →200+ sources integrated with standardized contracts
Lessons Learned
"The hardest part wasn't technology—it was getting 200 source teams to agree on data contracts. Spent 40% of the project on organizational alignment, and it was the best investment we made."