How to parameterize Delta Live Tables and import reusable functions with working code

Canadian Data Guy
4 min readDec 13, 2022
Photo by Roberto Nickson on Unsplash

This blog will discuss passing custom parameters to a Delta Live Tables (DLT) pipeline. Furthermore, we will discuss importing functions defined in other files or locations. You can import files from the current directory or a specified location using sys.path.append().

Update: As of December 2022, you can directly import files if the reusable_functions.py file exists in the same repository by just using the import command, which is the preferred approach. However, in case these reusable_functions.py file exists outside the repository, you can take the sys.path.append() approach mentioned below.

Overall, this a 4-step process:

  1. Create a reusable_functions.py file
  2. Add code to receive the DLT parameters
  3. Append the path to reusable_functions.py file and import the functions in the notebook
  4. Create a DLT pipeline and set/pass parameters

1. Create a reusable_functions.py file

Create a reusable function in a Python File (not Notebook), so we can import it later. Let’s call the file ‘reusable_functions.py’ below and place it in a path. Please make sure to note the absolute path of the folder where this file…

--

--

Canadian Data Guy

https://canadiandataguy.com | Data Engineering & Streaming @ Databricks | Ex Amazon/AWS | All Opinions Are My Own