PinnedCanadian Data GuyUsing Spark Streaming to merge/upsert data into a Delta Lake with working codeThis blog will discuss how to read from a Spark Streaming and merge/upsert data into a Delta Lake. We will also optimize/cluster data of…·4 min read·Oct 12, 2022--2--2
PinnedCanadian Data GuySpark Streaming Best Practices-A bare minimum checklist for Beginners and Advanced UsersMost good things in life come with a nuance. While learning Streaming a few years ago, I spent hours searching for best practices. However…·4 min read·Oct 27, 2022--1--1
PinnedCanadian Data GuyHow to parameterize Delta Live Tables and import reusable functions with working codeThis blog will discuss passing custom parameters to a Delta Live Tables (DLT) pipeline. Furthermore, we will discuss importing functions…·4 min read·Dec 13, 2022----
PinnedCanadian Data GuyHow to write your first Spark Stream Batch Join with working codeWhen I started learning about Spark Streaming, I could not find enough code/material which could kick-start my journey and build my…·4 min read·Jan 26, 2023----
Canadian Data GuyMerging Multiple Data Streams with Delta Live Tables: Kafka, Kinesis, and DeltaIntroduction·5 min read·1 hour ago----
Canadian Data GuyNeed for Speed: Benchmarking the Best Tools for Kafka to Delta IngestionIntroduction·7 min read·Jun 18, 2024----
Canadian Data GuySynthetic Data Made Simple: Generating and Streaming Custom-Sized Data to KafkaIntroduction·5 min read·Jun 16, 2024----
Canadian Data GuyLearnings from the Field: How to Give Your Spark Streaming Jobs a 15x Speed Boost Using the…Introduction:·4 min read·Mar 3, 2024----
Canadian Data GuyUnderstanding Delta Lake: A Technical Deep DiveDelta Lake is a powerful open-source storage layer that brings ACID transactions, scalable metadata handling, and unified batch and…·3 min read·Feb 27, 2024----
Canadian Data GuyStreaming Any File Type with Autoloader in Databricks: A Working GuideSpark Streaming has emerged as a dominant force as a streaming framework, known for its scalable, high-throughput, and fault-tolerant…·4 min read·Jan 4, 2024----