A Productive Life: How to Parallelize Code Execution in Python

Canadian Data Guy
4 min readApr 23

Asynchronous programming has become increasingly popular in recent years, especially in web development, where it is used to build high-performance, scalable applications. Python has built-in support for asynchronous programming through the asyncio module, which provides a powerful framework for writing asynchronous code.

In this blog post, we will explore the asyncio module in Python 3.10 and learn how to run tasks in parallel using the new features introduced in this version. We will explore 3 examples here:

Index:

· Example 1: Asyncio Tasks / create_task()
· Example 2: Running Tasks in Parallel
· Example 3: Running tasks in parallel with a loop
· Conclusion
· Footnote:

Example 1: Asyncio Tasks / create_task()

In asyncio, a task is a unit of work that is scheduled to run on the event loop. Tasks are created from coroutines, which are functions that are defined using the async def syntax and that can suspend their execution using the await keyword.

To create a task, we use the asyncio.create_task() function, which takes a coroutine as its argument and returns a Task object. We can then schedule the task to run on the event loop using the await keyword.

Here’s an example:

import asyncio

async def function_which_will_run_in_parallel():
# Add what you want to do here
print('function_which_will_run_in_parallel completed')

# Orchestrate function
async def main():
task = asyncio.create_task(function_which_will_run_in_parallel())
await task

asyncio.run(main())

If asyncio.run(main()) command does not work, replace it with await main()

In this example, we define a simple function_which_will_run_in_parallel() that waits for one second and then prints a message. In the main() function, we create a Task object using asyncio.create_task() and pass it the function.

We then await the completion of the task using await task. When we run the main() using asyncio.run(), the Task object is created and scheduled on the event loop, which runs the function_which_will_run_in_parallel() function asynchronously. Once the function_which_will_run_in_parallel() function is complete, the Task object is marked as…

Canadian Data Guy

https://canadiandataguy.com | Data Engineering & Streaming @ Databricks | Ex Amazon/AWS | All Opinions Are My Own