Batch vs. Streaming: When Seconds Justify Continuous Processing
Choose batch when throughput, cost and large-scale reprocessing matter most, and choose streaming when seconds or milliseconds of latency drive the business outcome. Batch latencies commonly run from minutes to hours or days, while streaming aims for seconds or milliseconds. Many production systems blend both, and unified engines blur the line. We deliver all three patterns.
TL;DR:
Bounded batch inputs produce repeatable reruns and straightforward backfills; streaming requires lateness rules, watermarks, checkpointing, idempotent sinks, and monitoring for lag and state growth.
Choose Lambda for audited results and a fast view, Kappa when retained logs support replay, or a hybrid that streams only urgent outputs.
Spark Structured Streaming can reach 100 milliseconds by default and 1 millisecond in Continuous Processing, although the modes provide different guarantees.
Before launch, test captured production data with shuffled replays, compare windowed outputs against batch results, measure tail latency, and build backfill before an incident.
Table of Contents
How do batch and streaming compare side by side?
The two models differ less in what they compute than in when they commit to an answer. Batch groups data into large, scheduled chunks and optimises for throughput. Streaming processes records incrementally as they arrive and optimises for latency. The practical differences fall along several dimensions:
Latency: batch answers arrive minutes to days after the data, streaming answers in seconds or milliseconds. A nightly revenue report tolerates the delay; a fraud check on a card payment does not.
Throughput: batch engines read large volumes in bulk and keep hardware busy, so they move more data per unit of compute. A reconciliation across extensive ledger history is a natural fit.
Cost profile: batch clusters can start, finish and shut down, so you pay for the run. Streaming clusters stay provisioned around the clock, which suits always-on monitoring but costs more when nobody needs the freshness.
Operational complexity: a failed batch job is rerun. A streaming job has no natural end, so you monitor lag, state size and checkpoints continuously. On-call load for a streaming pipeline is rarely the same as for a nightly job.
Determinism and semantics: bounded input gives repeatable output, so a rerun produces the same result. Streaming results depend on arrival order, windows and late data, which means you must define what “correct” means up front.
State handling and reprocessing: batch recomputes from source, which makes backfills straightforward. Streaming keeps running state for aggregations and joins, and a bug fix often means replaying history through the new logic.
Read the rows as constraints rather than scores. If your latency requirement is measured in hours, the streaming column buys you nothing except operational burden. If an operations team must react to an event before it ages, no amount of batch tuning closes the gap. Teams with edge devices face an extreme version of this choice, as edge versus cloud analytics shows when decisions must happen close to the source.
Why is streaming harder to engineer correctly?
Streaming engines trade throughput efficiency for lower latency. Continuous state and acknowledgement mechanisms add overhead, and the Spark documentation is blunt that there is no free lunch: choose streaming only after an honest assessment of whether low latency is genuinely required.
The harder problem is time. Event time is when something happened at the source. Processing time is when your system sees it. In batch, the data is already complete, so the distinction rarely bites. In streaming, records arrive late and out of order, and the engine must decide when a window is complete. Event-time windows, watermarks and triggers answer that question, and because streams are generally unbounded, every aggregate must be scoped by a window. You cannot count an infinite stream.
Stateful operations raise the stakes. Aggregations and joins hold data in memory or in a state backend, and watermarks govern both late data and state cleanup. Concepts like these have no counterpart in a simple bounded job. The engineering practices that follow are consistent:
Define a lateness tolerance per stream and write down what happens to records beyond it, whether dropped, side-output or reconciled by a batch correction.
Enable checkpointing and choose a state backend sized for peak key cardinality, not the average.
Expire state with watermarks or time-to-live settings so memory does not grow without limit.
Make sinks idempotent, so a replay after failure does not double-count.
Track tail latency and consumer lag, not just averages.
Pro Tip: Test with deliberately shuffled input. Replay captured events with random delays and swapped ordering, then compare the windowed output with a batch run over the same data. Any difference is a watermark or lateness bug.
Google’s guidance on streaming pipelines makes the same point: moving from batch to stream means designing for non-determinism, or analytics get corrupted by out-of-order arrivals.
What do Lambda, Kappa and hybrid patterns mean in practice?
Lambda and Kappa are two architectures described in a fundamentals paper. Lambda keeps a batch layer as the source of truth alongside a speed layer for fresh results. It delivers high accuracy, but you maintain two code paths that must agree. Kappa treats all data as a stream and replays history through the same logic. That simplifies the stack but increases state demands on the streaming engine.
The same source notes that Lambda often shows a bimodal latency distribution, while Kappa tends to show consistent latency but needs resources scaled to the data segment processed per iteration.
A practical way to pick:
Choose Lambda when a trusted, auditable batch result is non-negotiable and a fast approximate view is a bonus, as in regulated reporting.
Choose Kappa when your team can run a streaming engine confidently, the event log is retained long enough to replay, and one codebase matters more than raw batch efficiency.
Choose a pragmatic hybrid when only a few outputs need freshness: stream those, batch everything else, and avoid duplicating logic where you can.
Jimmy Lin’s analysis of the Lambda and Kappa debate frames the real question as matching business need to the latency, throughput and cost spectrum. Our own lakehouse guidance covers the storage side of replay and audit readiness.
How do Spark, Flink and Beam handle streaming?
The frameworks express the same ideas differently:
Apache Spark Structured Streaming: treats a live stream as a continuously appended table. The default micro-batch engine reaches latencies as low as 100 milliseconds, with a Continuous Processing mode that can reach 1 millisecond under different guarantees. It lets you write streaming logic with the same model as batch.
Apache Flink: built event-time first, with windows, watermarks and triggers as core primitives and strong support for complex stateful processing. It suits pipelines where correctness under disorder is the main concern.
Apache Beam: a unified model with bounded and unbounded collections and transforms that apply to both. Runners execute the pipeline, so the same code can target different engines.
Unified frameworks mainly improve developer productivity. They do not make a streaming job match a specialised batch job on raw throughput. Whatever you pick, evaluate the operational features before the API:
Delivery guarantees: exactly-once versus at-least-once, and what your sinks support.
Checkpointing frequency and recovery time.
State backend options and how state is cleaned up.
Monitoring for lag, watermark progress and backpressure.
How should you decide between batch, streaming and hybrid?
Work through the questions in order, because each narrows the next.
Fix the SLA. State the required latency and the acceptable staleness in business terms. If a ten-minute delay changes nothing, stop here and choose batch.
Measure the data. Check real velocity, burstiness and how out of order events arrive.
Cost the options. Compare an always-on streaming cluster with scheduled batch compute for the same outputs.
Check operations. Be honest about team skills, on-call capacity and monitoring readiness.
Plan replay. Decide how often you will backfill and how the pipeline reprocesses history.
Then validate before committing:
Prototype with micro-batches on a captured slice of production data.
Measure tail latency, not the median.
Test watermarks and late arrivals using shuffled replays.
Build the backfill mechanism before launch, not after the first incident.
Scheduling and backfill details are covered in our orchestration guide. For a case where documents arrive continuously and need quick handling, see our finance document processing example.
Why we favour the simplest pipeline that meets the SLA
Teams reach for streaming because it sounds modern, then pay for freshness nobody uses. Our rule of thumb is the opposite: take the simplest system that meets the SLA and allows deterministic reprocessing. Start with micro-batches, measure, and move to true streaming only when the business value of lower latency is demonstrable. When you present the choice to stakeholders, translate engineering consequences into cost and risk: what an hour of staleness costs, what an always-on cluster costs, and what a wrong windowed number costs.
— Thomas Samuel
We build and run production batch and streaming pipelines
If you need these pipelines in production rather than on a whiteboard, we cover the full path. Our data and platform engineering team designs and builds the pipelines, our deployment and MLOps work takes them live, and managed AI operations with ongoing optimisation keep them healthy afterwards. Our team owns strategy, build and operation, so there is no handover between design and run, and governance and continuity stay intact. To start with an assessment or a pilot, see all our services and tell us your latency target.
FAQ
What is the main difference between batch and streaming?
Batch processes data in large, scheduled chunks and optimises for throughput. Streaming processes records incrementally as they arrive and optimises for latency. The Microsoft Learn overview describes batch latencies of minutes to hours or days against seconds or milliseconds for streaming.
When should I use batch processing?
Use batch when results can wait, data volumes are large and you want cheap, repeatable reprocessing. Nightly reporting, periodic reconciliation and model training are typical cases. Bounded input gives deterministic reruns, which simplifies audit and backfill.
Is streaming always better for real-time analytics?
No. Streaming adds state management, event-time handling and always-on infrastructure, and Spark’s documentation advises confirming that low latency is genuinely required first. If the business can tolerate minutes of delay, micro-batches or scheduled batch are usually simpler and cheaper.
Can one framework handle both batch and streaming?
Yes. Spark Structured Streaming and Apache Beam let you express batch and stream computations with a similar model. They improve developer productivity, but streaming still needs watermarks and state cleanup that bounded jobs do not.
Sources
Recommended