When working with data, selecting the right tools can make all the difference in efficiency and performance. In this post, we’ll explore two popular Python libraries—Pandas and Polars—and compare their performance on common data operations using the Covertype dataset from scikit-learn. Pandas, a long-standing favorite in the data science community, offers ease of use and versatility, while Polars, a newer library built in Rust, is rapidly gaining traction for its speed and memory efficiency, particularly with larger datasets.

Setting Up the Environment

For our benchmarks, we’ll be using the Covertype dataset from the UCI Machine Learning Repository, which is conveniently available through scikit-learn. This dataset is both large enough to showcase performance differences and represents a real-world machine learning task.

Dataset Details

• Size: 581,012 instances (rows) and 54 features (columns)
• Task: Predicting forest cover type from cartographic variables
• Features: Include elevation, aspect, slope, distance to hydrology, wilderness area, soil type, and more
• Target: 7 different forest cover types (e.g., Spruce/Fir, Lodgepole Pine, Ponderosa Pine)

First, let’s import the necessary libraries and load our dataset:

import pandas as pd
import polars as pl
from sklearn.datasets import fetch_covtype
import time
import numpy as np

# Fetch the dataset
covtype_data = fetch_covtype()
X = covtype_data.data
y = covtype_data.target

# Convert to Pandas DataFrame
df_pandas = pd.DataFrame(X, columns=[f'feature_{i}' for i in range(X.shape[1])])
df_pandas['target'] = y

# Convert to Polars DataFrame
df_polars = pl.DataFrame(df_pandas)

# Print shapes to confirm they're the same
print("Pandas DataFrame shape:", df_pandas.shape)
print("Polars DataFrame shape:", df_polars.shape)

Output:

Pandas DataFrame shape: (581012, 55)
Polars DataFrame shape: (581012, 55)

Now that we have confirmed that both DataFrames have the same shape, let’s proceed with our performance comparisons.

1. Filtering Data

Let’s filter rows where ‘feature_0’ is greater than its mean value.

# Pandas
start_time = time.time()
filtered_pandas = df_pandas[df_pandas['feature_0'] > df_pandas['feature_0'].mean()]
pandas_time = time.time() - start_time

# Polars
start_time = time.time()
filtered_polars = df_polars.filter(pl.col('feature_0') > pl.col('feature_0').mean())
polars_time = time.time() - start_time

print(f"Pandas filtering time: {pandas_time:.4f} seconds")
print(f"Polars filtering time: {polars_time:.4f} seconds")
print(f"Polars is {pandas_time/polars_time:.2f}x faster than Pandas")

Results:

Pandas filtering time: 0.0741 seconds
Polars filtering time: 0.0183 seconds
Polars is 4.05x faster than Pandas

Polars outperforms Pandas in this filtering operation, being about 4.05 times faster. This is due to Polars’ efficient memory management and parallel execution capabilities.

2. Aggregation

Let’s calculate the mean of all numeric columns.

# Pandas
start_time = time.time()
mean_pandas = df_pandas.drop('target', axis=1).mean()
pandas_time = time.time() - start_time

# Polars
start_time = time.time()
mean_polars = df_polars.select(pl.all().exclude('target')).mean()
polars_time = time.time() - start_time

print(f"Pandas aggregation time: {pandas_time:.4f} seconds")
print(f"Polars aggregation time: {polars_time:.4f} seconds")
print(f"Polars is {pandas_time/polars_time:.2f}x faster than Pandas")

Results:

Pandas aggregation time: 0.1863 seconds
Polars aggregation time: 0.0083 seconds
Polars is 22.32x faster than Pandas

Polars is about 22.32 times faster than Pandas for this aggregation operation. This significant difference showcases Polars’ strength in performing calculations across multiple columns.

3. GroupBy Operation

Let’s group by the ‘target’ column and calculate the mean of all other columns.

# Pandas
start_time = time.time()
grouped_pandas = df_pandas.groupby('target').mean()
pandas_time = time.time() - start_time

# Polars
start_time = time.time()
grouped_polars = df_polars.group_by('target').agg(pl.all().exclude('target').mean())
polars_time = time.time() - start_time

print(f"Pandas groupby time: {pandas_time:.4f} seconds")
print(f"Polars groupby time: {polars_time:.4f} seconds")
print(f"Polars is {pandas_time/polars_time:.2f}x faster than Pandas")

Results:

Pandas groupby time: 0.0873 seconds
Polars groupby time: 0.0106 seconds
Polars is 8.23x faster than Pandas

Polars is about 8.23 times faster than Pandas for this groupby operation. Polars’ optimization strategies make a significant impact, especially with larger datasets like this one.

4. Sorting

Let’s sort the dataset by ‘feature_0’ in descending order.

# Pandas
start_time = time.time()
sorted_pandas = df_pandas.sort_values('feature_0', ascending=False)
pandas_time = time.time() - start_time

# Polars
start_time = time.time()
sorted_polars = df_polars.sort('feature_0', descending=True)
polars_time = time.time() - start_time

print(f"Pandas sorting time: {pandas_time:.4f} seconds")
print(f"Polars sorting time: {polars_time:.4f} seconds")
print(f"Polars is {pandas_time/polars_time:.2f}x faster than Pandas")

Results:

Pandas sorting time: 0.2027 seconds
Polars sorting time: 0.0656 seconds
Polars is 3.09x faster than Pandas

Polars demonstrates its efficiency in sorting operations, being about 3.09 times faster than Pandas. This performance gain can be significant when working with large datasets that require frequent sorting.

5. Feature Engineering

Let’s perform a common feature engineering task: calculating the z-score (standard score) for each feature. This standardization technique is often used in machine learning to normalize features to a common scale.

# Pandas
start_time = time.time()
pandas_zscore = df_pandas.copy()
for column in pandas_zscore.columns[:-1]:  # Exclude the target column
    pandas_zscore[f'{column}_zscore'] = (pandas_zscore[column] - pandas_zscore[column].mean()) / pandas_zscore[column].std()
pandas_time = time.time() - start_time

# Polars
start_time = time.time()
feature_cols = [f'feature_{i}' for i in range(54)]
polars_zscore = df_polars.with_columns([
    ((pl.col(col) - pl.col(col).mean()) / pl.col(col).std()).alias(f'{col}_zscore')
    for col in feature_cols
])
polars_time = time.time() - start_time

print(f"Pandas z-score calculation time: {pandas_time:.4f} seconds")
print(f"Polars z-score calculation time: {polars_time:.4f} seconds")
print(f"Polars is {pandas_time/polars_time:.2f}x faster than Pandas")

Results:

Pandas z-score calculation time: 0.5154 seconds
Polars z-score calculation time: 0.0919 seconds
Polars is 5.61x faster than Pandas

Polars demonstrates superior performance in this feature engineering task, being about 5.61 times faster than Pandas. This efficiency can be particularly beneficial when working with large datasets and complex feature engineering pipelines, which are common in machine learning workflows.

Summary of Benchmarks

Operation	Pandas (seconds)	Polars (seconds)	Speed Improvement
Filtering	0.0741	0.0183	4.05x
Aggregation	0.1863	0.0083	22.32x
GroupBy	0.0873	0.0106	8.23x
Sorting	0.2027	0.0656	3.09x
Feature Engineering (Z-score)	0.5154	0.0919	5.61x

Conclusion

In our benchmarks using the Covertype dataset, Polars consistently outperformed Pandas across all five common data operations. The performance difference was particularly noticeable in aggregation operations, where Polars was over 22 times faster than Pandas. GroupBy operations, feature engineering, and filtering also showed significant improvements with Polars.

However, it’s important to note that Pandas remains an excellent choice for many scenarios:

1. Ecosystem Integration: Pandas has a vast ecosystem of compatible libraries and tools.
2. Familiarity: Many data practitioners are already familiar with Pandas, making it easier to use and understand.
3. Documentation: Pandas has extensive documentation and community support.

Polars shines when:

1. Working with Large Datasets: As we’ve seen, Polars’ performance advantage becomes more pronounced with larger datasets.
2. Performance-Critical Applications: When speed is a top priority, Polars can significantly reduce processing time.
3. Memory-Constrained Environments: Polars is generally more memory-efficient than Pandas.

For any data practitioner or machine learning engineer, being versatile with both libraries is highly beneficial. Pandas is an excellent starting point due to its user-friendly nature and widespread adoption, but Polars should be your go-to when performance and efficiency are critical, especially with large-scale datasets.

Ultimately, the right tool depends on the specific demands of your project. We recommend conducting your own benchmarks to evaluate how each library performs on your datasets and infrastructure, enabling you to make the most informed decision for your workflow.

Note: The benchmarks in this post were run on a specific hardware configuration. Performance may vary depending on your system specifications and the versions of Pandas and Polars used. Always test on your own setup for the most accurate results.