Summary: Apache Spark 4.3, announced on November 5, 2026 and available in Databricks Runtime 19, extends Real-Time Mode to stateful operators — streaming aggregation, deduplication, and custom stateful processing — with sub-150 ms P99 end-to-end latency on three benchmark workloads. For builders, the interesting part is that supported stateful pipelines may no longer need a separate streaming engine, and the release also ships SCD Type 2 Auto CDC and opt-in small-data optimizations aimed at the run-inspect-refine loop of LLM-assisted development. The full picture comes from the Databricks engineering announcement; note that the collected text of that page is truncated partway through the SCD Type 2 example, so details beyond that point aren’t covered here.
What stateful Real-Time Mode actually gives you
Fraud scoring, personalization, and device monitoring all need two things at once: low latency and memory across events. Until now, Real-Time Mode handled stateless work; Spark 4.3 adds streaming aggregation, dropDuplicates, and Scala/Java transformWithState (SPARK-58635, SPARK-57228), built on Structured Streaming’s existing state store and checkpointing rather than a new programming model.
The numbers in SPARK-58884: dropDuplicates hit 136 ms P99 end-to-end, window aggregation 139 ms, and transformWithState 141 ms — all under the 150 ms bar. The Databricks authors are careful to frame this correctly: these are “measurements of the three tested workloads, not a latency guarantee for every pipeline.” There are no real-world production figures in the source, so treat the benchmarks as a ceiling-of-plausibility check, not a migration business case.
The payoff if it holds for your workload: one open engine for batch, analytics, AI, and now stateful low-latency streaming. That’s a stack consolidation decision, not just a tuning one.
The mechanism: pipelined shuffles replace blocking ones
This is the part worth understanding even if you never touch streaming. Traditional Spark shuffles are blocking — the producer stage fully materializes its output before the consumer starts. That cannot support continuous flow between long-running tasks, so Spark 4.3 adds native concurrent stage scheduling and pipelined shuffles (SPARK-54699, SPARK-58076, SPARK-56664).
The concrete change is PipelinedShuffleDependency (SPARK-58185), a new dependency type whose downstream stages can start reading while producers are still running. The scheduler treats stages connected this way as a pipelined group — one coordinated unit that is scheduled together and, importantly, fails together (SPARK-58263, SPARK-58398). A failure in any stage takes down the whole group. Because all stages run concurrently, the group is only admitted when enough task slots exist for all of its tasks; otherwise Spark fails fast rather than hanging.
Routing is done by dependency type: a regular ShuffleDependency still goes through a blocking manager (SortShuffleManager), while a pipelined dependency uses StreamingShuffleManager (SPARK-57141), which makes output available as it’s produced. The Databricks team positions this as generic engine capability, not one-off streaming plumbing — future workloads could move past stage-at-a-time execution too. The cost side: pipelined groups are hungrier on task slots up front, and coarse-grained failure is the trade you accept for concurrent stages.
Spark 4.3 also shifts to a quarterly feature-release cadence, down from every six months — useful to know when planning upgrades, since features will land in smaller increments.
Project Feather: cheaper iteration on small data
LLM-assisted development loops — generate a transformation, run it on a sample, inspect, refine — often process a few rows, yet scheduling and shuffle overhead can dwarf the compute. Project Feather targets exactly that fixed overhead.
Spark 4.3 introduces opt-in, shuffle-free single-task execution for eligible small, single-partition fragments (SPARK-57851): when a scan reports its output is already one partition, supported downstream operations skip the Exchange. Default eligibility limits are tight — qualifying file scans read one nonempty small file in a single partition, and qualifying local in-memory relations hold up to 1,000 rows. Supported shapes include scans, aggregations, distinct, windows, sorts, and limits.
There’s also optional Arrow-native Dataset caching (SPARK-57268), storing cached batches in Arrow format with compression and column statistics. It’s off by default; enable it with spark.sql.cache.serializer=org.apache.spark.sql.execution.columnar.ArrowCachedBatchSerializer at application startup. Broader in-process local repartitioning is explicitly deferred to the next release (SPARK-57399), and the source gives no performance numbers for Feather — so benchmark your own notebooks rather than assuming wins.
SCD Type 2 in Auto CDC
Spark 4.3 adds end-to-end SCD Type 2 support to open-source Auto CDC (SPARK-56249), exposed through Python, SQL, and Spark Connect (SPARK-58321, SPARK-58247, SPARK-58261). Instead of overwriting the latest value, every unique key becomes a sequence of records with __START_AT and __END_AT validity columns — so you can answer both “where does this customer live now” and “which region were they in when the order shipped” without hand-written merge logic. The collected source text cuts off mid-example here, so I can’t confirm details of the API surface beyond what’s stated above.
The release also rounds out SQL and connectivity: VARIANT transformations, JSON_TABLE, temporal joins, a nanosecond timestamp preview, plus Spark Connect observability and JDBC improvements — though the truncated text limits how much detail is verifiable.
What to check next
If you maintain a separate streaming engine purely for latency, prototype one stateful pipeline (a windowed aggregation or dedup job) in Real-Time Mode under Databricks Runtime 19 and measure P99 against your current system — with your data volumes, not the benchmarks’. Second, if your team lives in the generate-run-inspect loop with coding assistants, try Feather’s single-task path and the Arrow cache on a typical exploration notebook, respecting the 1,000-row and single-file limits, and diff the plan to confirm the Exchange actually disappears. Third, if you’re hand-rolling SCD 2 merge logic, the Auto CDC path in Python or SQL is worth a spike. And if you’re tracking where Databricks is headed as a company while adopting its runtime, see our earlier coverage of its Coatue-led round.
Sources
View the original post on X