Skip to main content

Data Processing

Data processing is the layer of a data platform that transforms raw, often unstructured, source data into clean, reliable, and analytically useful datasets. In a modern architecture, processing sits between ingestion and serving — consuming events and records from message queues, databases, and object stores, and producing curated tables, aggregations, and features that feed dashboards, machine learning models, and operational applications.

This section of BigDataDevPro covers the engines, patterns, and optimization techniques that make large-scale data processing possible. It focuses on batch processing as the foundational workload, while acknowledging the convergence with streaming that blurs the boundary in frameworks like Spark Structured Streaming. The emphasis throughout is on understanding how distributed processing works, not just what API to call.

What Is Data Processing?​

In the context of data engineering, processing refers to the systematic transformation of data using computational resources. This goes beyond simple SQL queries against a transactional database. Distributed data processing operates across clusters of machines, parallelizing work to handle terabytes and petabytes that exceed the capacity of a single node.

Key aspects:

  • Batch processing: Computations over bounded, finite datasets, typically scheduled and run periodically. Use cases include nightly ETL, data warehouse loading, and historical backfills.
  • Distributed processing: Dividing logic and data across multiple machines, coordinating execution, and handling partial failures.
  • Parallel computation: Splitting a single job into many independent tasks that execute concurrently on different CPU cores or nodes.
  • Data transformation: The core activity — cleaning malformed records, joining disparate sources, applying business rules, aggregating metrics, and enforcing schema.

Distributed data processing differs from application processing (which serves user requests with low latency and strong consistency) and from database processing (which operates within the tight transactional boundaries of a single DBMS). It is designed for analytical workloads: high throughput, read‑heavy, and tolerant of higher latency in exchange for scale.

Data Processing in Modern Data Platforms​

Processing engines are the compute backbone of an enterprise data architecture. A typical modern platform combines multiple processing patterns:

  • Data sources (application databases, event streams, log files) feed raw data into a landing zone.
  • Message systems like Kafka buffer events and decouple producers from consumers.
  • Processing engines — Spark, Trino, Flink — consume raw data, execute transformations, and write results to storage.
  • Storage layers — object stores (S3, ADLS, GCS) for data lakes, and open table formats (Iceberg, Delta) for lakehouse tables — hold data at various stages of refinement.
  • Analytics systems and AI/ML workloads then read processed data for querying, reporting, model training, and inference.

Common architectural patterns include:

  • ETL/ELT pipelines where processing engines run heavy transformations before (ETL) or after (ELT) data lands in the target warehouse or lakehouse.
  • Medallion architecture with bronze, silver, and gold layers, each the output of successive processing jobs that increase structure and quality.
  • Multi-engine lakehouse where Spark handles batch transformation, Trino serves interactive SQL, and Flink handles streaming updates, all operating on the same table formats and object storage.

Batch Data Processing​

Batch processing remains the workhorse of data platforms. It is the most predictable, cost‑effective way to perform large‑scale transformations when sub‑minute latency is not required.

Batch workloads are characterized by:

  • Scheduled execution: Jobs triggered by time (daily, hourly) or by upstream data availability.
  • High throughput over low latency: Optimized to process millions or billions of records per job, not single records.
  • Idempotency and reprocessability: The ability to rerun a job and produce the same result, essential for backfilling and error recovery.
  • Historical data processing: Re‑computing metrics over months or years of data when logic changes.

Typical batch use cases include daily sales aggregations, data warehouse loading from operational databases, log processing for user behavior analysis, and the creation of training datasets for machine learning. Apache Spark dominates this category, but other engines like Hive on Tez and Trino also play roles in batch execution.

Apache Spark Overview​

Apache Spark is the de facto standard for distributed batch processing. It unified the fragmented Hadoop ecosystem by offering a fast, general‑purpose engine with high‑level APIs in Python, SQL, Scala, and Java, all sharing the same optimizer and execution runtime.

Spark’s architecture consists of:

  • Driver: The process that contains the user’s main method, defines the job, and coordinates execution. It converts logical plans into a DAG of stages and tasks.
  • Executors: JVM processes on worker nodes that run individual tasks, store data in memory or disk, and report status back to the driver.
  • Cluster Manager: The system that allocates resources (YARN, Kubernetes, Spark Standalone). Spark requests containers or pods and launches executors within them.
  • Job: A complete computation triggered by an action (e.g., count(), save()), consisting of one or more stages.
  • Stage: A set of tasks that can be executed without a shuffle boundary. Stages are separated by operations that require data redistribution.
  • Task: The smallest unit of work — a computation on a single data partition executed by a single executor thread.

This decomposition allows Spark to scale from a single laptop to thousands of nodes, with the same programmatic interface.

Spark Data Processing Model​

Spark provides three main data abstractions, each with a distinct API and level of optimization:

  • RDD (Resilient Distributed Dataset): The original low‑level API. Provides fine‑grained control over data distribution and computation but requires manual optimization. Used today only for specialized workloads that cannot be expressed in higher‑level APIs.
  • DataFrame: A distributed collection of rows with named columns, conceptually equivalent to a table in a relational database. DataFrames benefit from Catalyst optimization and are the recommended abstraction for most batch processing.
  • Dataset: A type‑safe version of DataFrame available in Scala and Java. Combines the expressiveness of RDDs with the optimization of DataFrames. In Python, the DataFrame API effectively subsumes this.

Modern Spark applications overwhelmingly use DataFrames and Spark SQL. These APIs enable:

  • Lazy evaluation: Transformations (select, filter, join) build a logical plan; execution is deferred until an action forces materialization. This allows the optimizer to reorganize the plan for efficiency.
  • DAG execution: Spark builds a directed acyclic graph of stages. The Catalyst optimizer applies rule‑based and cost‑based optimizations to produce an efficient physical plan.
  • Transformations and Actions: Transformations are lazy (e.g., filter), while actions trigger computation (e.g., count, write).

This model abstracts distributed complexity while preserving the ability to tune performance at the level of partitions, caching, and join strategies.

Spark SQL and Query Processing​

Spark SQL bridges the gap between declarative SQL and distributed execution. It exposes a SQL interface on top of DataFrames, allowing users to query structured data using ANSI SQL and integrate seamlessly with Hive metastores.

The query processing pipeline:

  1. Parsing: SQL text or DataFrame operations are parsed into an unresolved logical plan.
  2. Analysis: Table and column references are resolved against the catalog.
  3. Logical optimization: Rule‑based optimizations (predicate pushdown, constant folding, projection pruning) improve the plan.
  4. Physical planning: The optimizer generates multiple physical execution strategies (e.g., different join algorithms) and selects the cheapest using cost estimates.
  5. Code generation: Tungsten, Spark’s execution engine, generates optimized Java bytecode for expression evaluation and in‑memory columnar processing, minimizing virtual function calls and leveraging CPU caches.

Understanding this pipeline is critical for reading query plans, interpreting Spark UI metrics, and writing efficient transformations. A well‑written SQL query can automatically benefit from partition pruning, broadcast joins, and vectorized reads — all without manual API tuning.

Distributed Processing Concepts​

Several low‑level mechanisms determine the performance of any distributed processing job. These are not Spark‑specific; they apply equally to Trino, Flink, and other MPP engines.

  • Partitioning: The division of data into chunks that can be processed in parallel. Good partitioning minimizes data movement and balances load across executors.
  • Parallelism: The number of concurrent tasks. Determined by the number of partitions and available cores. Too low underutilizes the cluster; too high causes scheduling overhead.
  • Shuffle: The redistribution of data across partitions, usually triggered by wide transformations (groupBy, join). Shuffles are expensive because they involve disk I/O and network transfer. Reducing and optimizing shuffles is the single most impactful performance lever.
  • Data locality: Placing computation on the node that stores the data to avoid network transfer. Spark’s scheduler prefers locality, but shuffle inevitably breaks it.
  • Serialization: Converting objects to byte streams for network transfer and disk spilling. Efficient serialization (Kryo, Tungsten binary format) reduces I/O and memory pressure.
  • Memory management: Balancing execution memory (used for shuffles, sorts, aggregations) and storage memory (used for caching DataFrames). Spilling to disk occurs when execution memory is exhausted.

Spark Performance Optimization​

Production Spark pipelines require deliberate tuning to meet SLAs and control costs. Optimization is not guesswork; it is the systematic reduction of I/O, shuffle, and CPU waste.

Common optimization areas include:

  • Partition sizing: Aim for partitions of 128 MB to 256 MB (compressed) to balance parallelism and task overhead. Use coalesce and repartition to adjust.
  • Broadcast joins: When one side of a join is small (below the broadcast threshold), Spark can send it to all executors, eliminating the shuffle entirely.
  • Caching and persistence: Storing intermediate DataFrames in memory or on disk to avoid recomputation across multiple actions. Effective when the same dataset is reused.
  • File format selection: Columnar formats (Parquet, ORC) with compression enable vectorized reads and predicate pushdown. Avoid text‑based formats for large datasets.
  • Shuffle reduction: Favor reduceByKey over groupByKey, use spark.sql.shuffle.partitions to control partition count, and enable Adaptive Query Execution (AQE).
  • Adaptive Query Execution (AQE): Dynamically optimizes query plans at runtime based on actual data statistics — coalescing small partitions, switching join strategies, and handling skew. Available from Spark 3.0+.
  • Data skew handling: When a few keys dominate a transformation, add a random salt to spread the work, then aggregate in two phases.

Effective optimization requires fluency with the Spark UI — reading DAG visualizations, stage timings, and shuffle read/write metrics to pinpoint bottlenecks.

Other Data Processing Technologies​

While Spark dominates batch processing, a modern platform often incorporates multiple engines, each optimized for specific access patterns.

Trino / Presto​

Trino (formerly PrestoSQL) is a distributed SQL query engine designed for interactive, sub‑second analytics. Unlike Spark, which is optimized for long‑running batch jobs, Trino excels at low‑latency queries across diverse data sources — object stores, relational databases, and streaming systems — all through a single SQL interface. Trino’s MPP architecture executes queries in‑memory, pipelining data between stages without writing intermediate results to disk, making it ideal for ad‑hoc exploration and BI dashboards.

Apache Hive​

Hive was the original SQL engine for Hadoop, translating SQL into MapReduce (and later Tez or Spark) jobs. While its batch‑oriented HiveServer2 and Hive Metastore remain widely used for cataloging, the execution engine has been largely superseded by Spark and Trino for transformation workloads. Hive’s legacy lies in the metastore standard it established, which now underpins lakehouse catalogs.

dbt (data build tool)​

dbt represents the rise of analytics engineering — treating SQL transformations as software. It does not execute transformations itself; rather, it orchestrates SQL statements against a data warehouse or lakehouse engine, adding testing, documentation, and lineage tracking. dbt brings CI/CD principles to data processing, enabling modular, version‑controlled transformation pipelines that complement Spark and Trino workloads.

Building Production Data Pipelines​

A processing job that works on a laptop is not production‑ready. Production data pipelines exhibit a set of operational characteristics that ensure they continue to deliver reliable data over time.

  • Reliability: Jobs must tolerate transient failures, retry gracefully, and produce correct results even when interrupted.
  • Scalability: The system must scale horizontally to accommodate growing data volumes without linear increases in runtime or cost.
  • Monitoring and observability: Metrics on record counts, lateness, and data quality anomalies must be surfaced, with alerts tied to SLOs.
  • Data quality: Validations — null checks, referential integrity, range constraints — are embedded in the pipeline, not left to downstream consumers.
  • Error handling and dead letter queues: Malformed records are quarantined, not silently dropped, allowing debugging and reprocessing.
  • Backfilling and reprocessing: Pipelines are designed to be idempotent and time‑travel aware, enabling re‑calculation of historical data without side effects.
  • Testing: Unit tests for transformation logic, integration tests for connector behavior, and schema contract tests are automated in CI.
  • Idempotency: Rerunning the same pipeline with the same inputs produces identical output, critical for exactly‑once semantics and operational simplicity.

Data Processing Learning Path​

The following sequence builds proficiency from conceptual understanding to production implementation:

  1. Understand Data Processing Fundamentals — grasp batch vs. streaming, ETL vs. ELT, and the role of processing in a platform.
  2. Learn SQL and Data Transformation — become fluent in window functions, aggregations, and complex joins; these will translate directly to Spark SQL.
  3. Understand Distributed Computing — internalize partitioning, shuffle, fault tolerance, and resource scheduling.
  4. Learn Apache Spark — start with DataFrames, work through the execution model, and practice reading Spark UI graphs.
  5. Master Spark SQL — use the Catalyst optimizer to your advantage; learn to read query plans and optimize SQL for distributed execution.
  6. Learn Performance Optimization — apply broadcast joins, manage partition sizes, handle skew, and enable AQE.
  7. Explore Distributed Query Engines — understand Trino and its role alongside Spark in a multi‑engine architecture.
  8. Build Production Data Pipelines — incorporate orchestration, monitoring, data quality checks, and CI/CD to move beyond ad‑hoc scripts.

The articles in this section are structured to follow this progression:

Connection to Other Sections​

Data Foundations — The Data Foundations section provides the conceptual groundwork for everything in Data Processing. Concepts like partitioning, file formats, and distributed computing principles are directly applied when tuning Spark jobs. A solid foundation in storage models and serialization ensures that processing decisions are architecturally sound.

Data Streaming — Batch processing and stream processing are two sides of the same coin. The Data Streaming section explores Kafka, Flink, and Spark Structured Streaming. Many organizations run batch and streaming pipelines on the same platform, and the processing concepts you learn here — state management, watermarks, checkpointing — carry over directly.

Data Architecture — Processing engines are components of a larger platform. The Data Architecture section shows how Spark, Trino, and dbt fit into lakehouse designs, medallion architectures, and data mesh implementations. Understanding processing is necessary, but it is through architecture that you decide which engine does what and where data should live.