Image by Editor | Perplexity

Introduction

Data scientists often work with large datasets, where the performance of data processing and numerical computation is critical. While Python is the language of choice for many in the data science, machine leanring and AI worlds due to its flexibility and rich ecosystem of libraries, standard loops, when implemented with the language, can offer slow performance by default. This bottleneck arises because Python is an interpreted, dynamically-typed language, which introduces significant overhead for each iteration.

This article will demonstrate a better way: loop optimization with Cython. This approach can dramatically speed up your Python code, and operates by compiling it into efficient C code.

The Inefficiency of Python Loops

In data science, we frequently need to iterate over large arrays or data structures to perform calculations. Consider the simple task of summing the elements of a large array: in pure Python, especially without leveraging optimized libraries like NumPy, this can be slow. The Python interpreter must check the type of each variable at every step, which adds up to a lot of wasted time when dealing with potentially millions of iterations.

Let’s look at a baseline Python function that sums elements in a list.

# baseline_python.py
def sum_list_python(data_list):
    s = 0
    for x in data_list:
        s += x
    return s

While simple, this function’s performance degrades significantly as the size of data_list increases. For performance-critical applications, this would lead to unacceptable outcomes.

Cython for a Performance Boost

Cython is a programming language that is also a superset of Python. It is designed to give C-like performance with code that is written mostly in Python. The key feature of Cython is its ability to support static type declarations. By telling Cython the data types of your variables beforehand, you eliminate Python’s dynamic type-checking overhead, allowing the code to be compiled into much faster C code.

To optimize our loop, we will create a Cython file with a .pyx extension. Inside this file, we will add type declarations to our variables using the cdef keyword.

Here is our loop rewritten in a file named cython_loop.pyx:

# cython_loop.pyx
def sum_list_cython(data_list):
    # Static type declarations
    cdef long long s = 0
    cdef int x

    for x in data_list:
        s += x
    return s

The cdef keyword declares s as a C-level long long integer and x as a C-level integer. This simple change allows Cython to generate highly optimized C code for the loop. You can find more information on the structure and syntax of Cython files, as well as details on compiling the file, in the previous post in this series.

Once compiled, you will be left with a module with a name such as cython_loop.so (on Linux) that you can import directly into your Python scripts.

Benchmarking the Performance

Now we can compare the execution speed of our original Python function with our new, optimized Cython function.

import timeit
from baseline_python import sum_list_python
from cython_loop import sum_list_cython

# Create a large list of numbers
large_list = list(range(10000000))

# Time the Python function
python_time = timeit.timeit('sum_list_python(large_list)', 
                            globals=globals(), 
                            number=10)

# Time the Cython function
cython_time = timeit.timeit('sum_list_cython(large_list)', 
                            globals=globals(), 
                            number=10)

print(f"Pure Python version: {python_time:.4f} seconds")
print(f"Cython version: {cython_time:.4f} seconds")

Sample output:

Pure Python version: 4.5213 seconds
Cython version: 0.8145 seconds

As you can see from this sample output, the Cython version is significantly faster. This speedup comes from converting the Python loop into a much more efficient C loop, all with but a few simple type declarations.

Wrapping Up

When your data science workflow is hampered by slow Python loops, Cython offers a way to optimize your code. By adding static types to your variables, you can compile your Python-like code into efficient C modules, resulting in sometimes dramatic performance improvements. This allows you to keep the high-level simplicity of Python while harnessing the speed of C for the parts of your code that need it most.