Data Architecture

Data architecture defines the blueprint for how an organization collects, stores, processes, and serves data. It is the strategic layer that connects business objectives with technology choices, ensuring that data platforms are not merely collections of tools but coherent, scalable, and governable systems. In an era where data fuels everything from operational dashboards to machine learning models, architecture is what separates reliable, cost‑effective platforms from fragile, ad‑hoc silos.

Enterprise data architecture has undergone a series of transformations. Organizations moved from monolithic data warehouses to Hadoop‑based data lakes, then to cloud‑native lakehouse architectures that combine the flexibility of lakes with the reliability of warehouses. Today’s architect must navigate batch and real‑time processing, decentralized data mesh principles, open table formats, and rigorous governance requirements — all while balancing cost, performance, and operational overhead.

This section of BigDataDevPro takes an architecture‑first view. It explains the design patterns, trade‑offs, and reference implementations behind modern data platforms. The articles in this section are not tutorials for specific products; they are decision guides for engineers and architects building platforms that will evolve with their organizations.

What Is Data Architecture?

Data architecture is the structural design of an organization’s data assets and the systems that manage them. It defines:

• Logical and physical data models: How data entities relate and how they are stored.
• Data flows: The movement of data from sources to consumers, including ingestion, transformation, and serving.
• Technology platforms: The selection and integration of storage, compute, and governance components.
• Non‑functional requirements: Scalability, reliability, security, and compliance constraints that shape implementation.

The role of a data architect is to align these elements with business goals. This involves making deliberate trade‑offs — accepting eventual consistency for global scalability, or choosing batch processing over streaming to reduce operational complexity — and documenting the rationale so that teams can evolve the platform without losing coherence.

Data architecture sits within the broader enterprise architecture, intersecting with application architecture (where data originates), infrastructure architecture (where it runs), and security architecture (how it is protected). A well‑designed data architecture enables self‑service access, reduces duplication, and accelerates time‑to‑insight.

Modern Data Platform Architecture

A modern data platform is typically decomposed into logical layers that separate concerns and enable independent scaling. The following diagram captures the canonical structure:

• Data sources: Operational databases (OLTP), application logs, third‑party APIs, IoT devices, and external data feeds. These produce structured, semi‑structured, and unstructured data.
• Ingestion layer: Tools and services that capture data from sources and deliver it reliably to the platform. This includes batch file uploads, Kafka for streaming, and CDC connectors for database replication.
• Processing layer: Distributed engines like Apache Spark, Trino, and Flink that clean, transform, join, and aggregate data. Processing may be scheduled (batch) or continuous (streaming), and often both coexist.
• Storage layer: The persistent repository for data at various stages of refinement. Modern platforms favor object storage (S3, ADLS, GCS) combined with open table formats (Iceberg, Delta Lake, Hudi) to provide a lakehouse: ACID transactions, schema enforcement, and efficient query performance on low‑cost storage.
• Analytics and applications layer: The consumers of processed data — BI tools, ad‑hoc SQL interfaces, machine learning training pipelines, reverse‑ETL to operational systems, and real‑time dashboards.
• Governance and operations: A horizontal layer that spans every other tier. It includes metadata catalogs, data lineage, quality monitoring, access control, and pipeline orchestration. Without strong governance, platforms become data swamps.

This layered model is not a strict physical topology. Cloud services often blur boundaries — a managed service like Databricks can handle ingestion, processing, and serving within a single workspace — but the logical separation helps architects reason about dependencies and failure domains.

Data Lake Architecture

A data lake is a centralized repository that stores raw data in its native format, typically on low‑cost object storage. Its defining characteristic is schema‑on‑read: data is ingested without pre‑defined structure, and consumers apply a schema at query time.

The primary motivation for data lakes has been cost and flexibility. Object storage is orders of magnitude cheaper than proprietary warehouse storage, and the ability to retain raw, untransformed data enables future reprocessing. Data lakes support both structured tables and unstructured files (logs, images, audio), making them suitable for exploratory data science and machine learning workloads.

In practice, lakes often organize data into zones:

• Raw zone: As‑ingested data, immutable and preserved for replay.
• Curated zone: Cleaned, validated, and maybe partitioned data, often in columnar formats.
• Refined zone: Business‑aggregated datasets, analogous to warehouse marts.

However, data lakes without strong governance can degrade into unmanageable swamps. Challenges include:

• Poor data quality and lack of schema enforcement.
• Metadata fragmentation, making data discovery difficult.
• Performance unpredictability due to lack of indexing and ACID guarantees.
• Complex permission management across heterogeneous files.

These limitations drove the emergence of the lakehouse.

Data Warehouse Architecture

A data warehouse is a structured repository optimized for high‑concurrency SQL queries on integrated, historical data. It relies on schema‑on‑write: data is modeled, transformed, and loaded into pre‑defined tables, ensuring consistency and query performance.

The dimensional modeling approach — fact tables for measurable events, dimension tables for descriptive context, organized in star or snowflake schemas — remains the foundation of warehouse design. Warehouses are purpose‑built for BI workloads: they provide fast aggregations, efficient joins, and mature security models.

Data warehouses are the right choice when:

• The query patterns are known and stable.
• Strict data quality and consistency are non‑negotiable.
• Concurrency demands are high, with hundreds of users running interactive reports.
• Storage and compute are tightly coupled in a managed service (e.g., Snowflake, BigQuery, Redshift), simplifying operations.

The historical downside of warehouses was the high cost of proprietary storage and the rigidity of ETL pipelines. As a result, organizations often kept a data lake for raw data and a warehouse for aggregated, business‑ready data — leading to dual‑system complexity.

Lakehouse Architecture

The lakehouse merges the best properties of data lakes and warehouses. It uses object storage as the durable foundation, overlaid with an open table format that provides ACID transactions, schema enforcement and evolution, time travel, and efficient indexing — all within a single storage layer.

The key enablers are:

• Apache Iceberg: A high‑performance table format with snapshot isolation, hidden partitioning, and broad engine support (Spark, Flink, Trino, Presto, Hive).
• Delta Lake: Deeply integrated with Spark, providing a transaction log, data skipping, and Z‑ordering.
• Apache Hudi: Focused on streaming ingestion, upserts, and incremental processing.

A lakehouse architecture eliminates the need to maintain separate lake and warehouse copies. Engines can read and write the same tables concurrently, and the storage is decoupled from compute, allowing elastic scaling. Medallion architectures (bronze, silver, gold) naturally map onto lakehouse tables, with each layer benefiting from ACID guarantees.

The lakehouse is now the dominant paradigm for new data platform construction. It offers the cost profile of object storage with the reliability of a warehouse, and it supports the full range of workloads — SQL analytics, data science, and streaming — on a unified platform.

Architecture	Storage	Schema	Transactions	Best For
Data Lake	Object storage (S3, ADLS)	Schema‑on‑read	None / eventual	Raw data, ML, exploration
Data Warehouse	Proprietary (block/columnar)	Schema‑on‑write	Full ACID	BI, structured analytics
Lakehouse	Object storage + open table format	Schema‑on‑read and enforcement	ACID via table format	Unified analytics, streaming, data science

Medallion Architecture

The medallion architecture (bronze, silver, gold) is a pattern for organizing data within a lakehouse. It provides a clear progression of data quality and structure:

• Bronze layer: Raw ingested data, often in the original format. Data is appended, never mutated. Acts as the source of truth and enables historical reprocessing.
• Silver layer: Cleaned, validated, and deduplicated data. Joins and standardizations are applied. Suitable for ad‑hoc analysis and exploratory queries.
• Gold layer: Business‑level aggregates, dimensions, and metrics. Optimized for consumption by BI tools and dashboards.

This pattern decouples ingestion from transformation, and transformation from serving. It allows teams to evolve business logic in the silver and gold layers without modifying the bronze landing zone, and it facilitates data quality checks at each transition.

Data Mesh Architecture

Data mesh is an organizational and architectural paradigm that addresses the scaling limitations of centralized data platforms. It applies product thinking and domain‑oriented ownership to analytical data.

Core principles:

• Domain ownership: Each business domain (marketing, logistics, finance) owns its data pipelines, quality, and serving.
• Data as a product: Domains publish data products that are discoverable, trustworthy, and self‑serviceable.
• Self‑serve data infrastructure: A platform team provides tools, standards, and automation, but domains own the data.
• Federated governance: Global policies (privacy, compliance) are defined centrally, but enforcement is integrated into domain workflows.

Data mesh is not a technology choice; it is an operating model. It reduces bottlenecks in centralized data teams but increases the need for strong platform engineering and governance. It is most appropriate for large organizations with diverse data sources and autonomous engineering teams. For smaller organizations, a centralized lakehouse often provides a better balance of simplicity and capability.

Cloud Data Platform Architecture

Cloud providers offer a rich set of managed services that accelerate data platform construction. The architecture remains conceptually the same, but the components are realized as cloud‑native services.

• Object storage: S3, ADLS Gen2, GCS provide the lakehouse foundation.
• Managed processing: AWS EMR, Databricks, Azure Synapse Spark, or Google Dataproc handle batch transformation. Serverless options like Athena or BigQuery provide SQL querying directly on lake data.
• Streaming: Amazon MSK, Azure Event Hubs, Google Pub/Sub for managed Kafka, with Kinesis or Dataflow as stream processing alternatives.
• Warehousing: Redshift, Synapse Dedicated Pool, BigQuery for high‑concurrency analytics, often querying lakehouse tables via external tables.
• Governance: AWS Glue Catalog, Azure Purview, Dataplex provide metadata management, lineage, and policy enforcement.

When designing cloud platforms, architects must weigh the trade‑offs between managed services (operational simplicity, less control) and self‑managed open‑source deployments (more flexibility, greater operational burden). A common approach is to standardize on object storage and open table formats while using managed compute engines, avoiding vendor lock‑in at the data layer.

Data Governance Architecture

Governance should be embedded into the architecture, not bolted on after deployment. A robust governance framework includes:

• Metadata management and data catalog: Centralized inventory of datasets, schemas, and business definitions. Tools like DataHub, Amundsen, or cloud‑native catalogs enable search and discovery.
• Data lineage: Tracking the origin and transformation history of data from source to consumption. Essential for impact analysis and debugging.
• Data quality: Automated checks for completeness, uniqueness, timeliness, and consistency at each pipeline stage. Quality dashboards and SLAs hold teams accountable.
• Access control: Fine‑grained permissions (column, row, table) enforced via IAM, Apache Ranger, or native table format security. Audit logging captures all data accesses.
• Compliance: Managing PII, data retention, and right‑to‑erasure requirements through classification, masking, and lifecycle policies.

Governance is not just about control; it enables self‑service. When users can trust that a certified dataset is accurate, fresh, and properly secured, they consume it with confidence.

Data Platform Scalability and Reliability

Architecting for scale means planning for growth in data volume, concurrent users, and workload diversity.

• Horizontal scalability: Every component — storage, compute, streaming — must scale out by adding nodes, not just up with bigger machines. Partitioning, consumer groups, and cluster autoscaling are standard techniques.
• Fault tolerance: Stateless processing can restart; stateful systems rely on checkpointing and replication. Multi‑AZ or multi‑region deployments protect against infrastructure failures.
• High availability: Redundant service instances, leader‑follower failover (Kafka, databases), and load‑balanced query endpoints eliminate single points of failure.
• Disaster recovery: Regular backups of metadata and catalogs, cross‑region replication of object storage, and the ability to replay ingestion from source logs.
• Cost optimization: Rightsizing compute, leveraging spot/transient instances, tiering storage, and monitoring resource usage to prevent runaway costs. Architectural decisions like file sizing and partition counts directly impact cost.
• Performance optimization: Proper partitioning, caching, materialized views, and query optimizer tuning are architectural responsibilities, not afterthoughts.

Reference Architecture Patterns

Common enterprise patterns provide starting points that can be adapted to specific needs.

Batch Analytics Platform

Ingestion via scheduled file uploads or periodic JDBC extractions → Raw data in object storage → Spark batch transformations (silver/gold) → data served via a lakehouse query engine (Trino, Spark SQL) or loaded into a warehouse. Governance via catalog and Airflow orchestration.

Real-Time Streaming Platform

Event sources → Kafka → Flink or Spark Structured Streaming for real‑time aggregation and enrichment → sink to low‑latency serving layers (Druid, Pinot, or lakehouse tables with frequent compaction) for operational dashboards and alerting.

Hybrid Data Platform

Combines batch and streaming in a unified architecture. CDC streams keep lakehouse tables up‑to‑date, while nightly batch jobs perform heavy reprocessing and recomputation of long‑window aggregates. A single orchestration layer manages both worlds, and governance spans all pipelines.

Data Architecture Learning Path

1. Understand data platform fundamentals: Grasp the role and boundaries of each layer — ingestion, storage, processing, serving, governance.
2. Learn data lake and warehouse patterns: Study their history, strengths, and limitations to understand why lakehouses emerged.
3. Master lakehouse architecture: Dive into Iceberg, Delta Lake, Hudi — their internals, trade‑offs, and how they enable modern platforms.
4. Learn data modeling: Dimensional modeling, medallion architecture, and how data products are structured for consumption.
5. Study streaming architectures: Integrate Kafka and Flink into platform designs, understanding how they coexist with batch.
6. Learn governance and security: Move beyond access control to catalog, lineage, and quality frameworks.
7. Design enterprise data platforms: Combine these elements into reference architectures that meet specific business requirements.

Recommended Articles

The articles in this section address each of these architectural concerns in depth:

• Modern Data Platform Architecture — A comprehensive walkthrough of the layered platform design with open‑source and cloud components.
• Designing a Data Lakehouse — How to build a lakehouse from scratch, covering storage, table formats, and multi‑engine access.
• Apache Iceberg Explained — Deep dive into Iceberg’s specification, partitioning, and snapshot mechanics.
• Delta Lake vs Iceberg vs Hudi — A comparative analysis of the three leading open table formats, including feature matrices and use case guidance.
• Medallion Architecture Explained — Designing bronze, silver, and gold layers with practical examples.
• Data Mesh vs Centralized Data Platform — A critical examination of the data mesh operating model versus centralized lakehouse approaches.
• Data Governance Fundamentals — Practical governance implementation: catalogs, lineage, quality, and compliance.
• Cloud‑Native Data Platform on AWS — A reference architecture using AWS services with architectural decisions explained.

Connection to Other Sections

Data Foundations — Architecture decisions are grounded in the fundamentals covered in Data Foundations. Storage models, partitioning, file formats, and distributed computing principles directly inform platform design. A weak foundation leads to brittle architectures.

Data Processing — The Data Processing section details the engines (Spark, Trino) that execute within the architecture. Architecture defines where and why processing happens; the processing section covers how it is implemented and optimized.

Data Streaming — Real‑time components must be architected as part of the whole. The Data Streaming section explores Kafka, Flink, and event‑driven patterns, which this section integrates into end‑to‑end platform designs.

Data Interview — Senior data engineering and architecture interviews test system design and architectural thinking. The Data Interview section provides practice with architecture case studies, questions on trade‑offs, and guidance on communicating design decisions — directly reinforcing what you learn here.