Merge Multiple Spark Streams Into A Delta Table with working code
This blog will discuss how to read from multiple Spark Streams and merge/upsert data into a single Delta Table. We will also optimize/cluster data of the delta table.
Overall, the process works in the following manner:
- Read data from a streaming source
- Use this special function foreachBatch. Using this we will call any user-defined function responsible for all the processing.
- Our user-defined function runs the Merge and Optimize over the…