PinnedCanadian Data GuyUsing Spark Streaming to merge/upsert data into a Delta Lake with working codeThis blog will discuss how to read from a Spark Streaming and merge/upsert data into a Delta Lake. We will also optimize/cluster data of…Oct 12, 20222Oct 12, 20222
PinnedCanadian Data GuySpark Streaming Best Practices-A bare minimum checklist for Beginners and Advanced UsersMost good things in life come with a nuance. While learning Streaming a few years ago, I spent hours searching for best practices. However…Oct 27, 20221Oct 27, 20221
PinnedCanadian Data GuyHow to parameterize Delta Live Tables and import reusable functions with working codeThis blog will discuss passing custom parameters to a Delta Live Tables (DLT) pipeline. Furthermore, we will discuss importing functions…Dec 13, 2022Dec 13, 2022
PinnedCanadian Data GuyHow to write your first Spark Stream Batch Join with working codeWhen I started learning about Spark Streaming, I could not find enough code/material which could kick-start my journey and build my…Jan 26, 2023Jan 26, 2023
Canadian Data GuyMerging Multiple Data Streams with Delta Live Tables: Kafka, Kinesis, and DeltaIntroduction2d ago2d ago
Canadian Data GuyNeed for Speed: Benchmarking the Best Tools for Kafka to Delta IngestionIntroductionJun 18Jun 18
Canadian Data GuySynthetic Data Made Simple: Generating and Streaming Custom-Sized Data to KafkaIntroductionJun 16Jun 16
Canadian Data GuyLearnings from the Field: How to Give Your Spark Streaming Jobs a 15x Speed Boost Using the…Introduction:Mar 3Mar 3
Canadian Data GuyUnderstanding Delta Lake: A Technical Deep DiveDelta Lake is a powerful open-source storage layer that brings ACID transactions, scalable metadata handling, and unified batch and…Feb 27Feb 27
Canadian Data GuyStreaming Any File Type with Autoloader in Databricks: A Working GuideSpark Streaming has emerged as a dominant force as a streaming framework, known for its scalable, high-throughput, and fault-tolerant…Jan 4Jan 4