Merge Multiple Spark Streams Into A Delta Table with working code

Canadian Data Guy
4 min readOct 13, 2022

This blog will discuss how to read from multiple Spark Streams and merge/upsert data into a single Delta Table. We will also optimize/cluster data of the delta table.

Overall, the process works in the following manner:

  1. Read data from a streaming source
  2. Use this special function foreachBatch. Using this we will call any user-defined function responsible for all the processing.
  3. Our user-defined function runs the Merge and Optimize over the…

--

--

Canadian Data Guy

https://canadiandataguy.com | Data Engineering & Streaming @ Databricks | Ex Amazon/AWS | All Opinions Are My Own