Image by Editor | Midjourney & Canva
Editor’s note: Statology Sprints are our condensed courses that fuse foundational stats material with expert-curated insights to supercharge your skills. Each sprint distills a set of related complex concepts into actionable tools, adds fresh perspectives, and skips the filler. Learning Polars can be hard. That’s why we’re here to help.
Learning statistics can be hard. It can be frustrating. And more than anything, it can be confusing. That’s why we’re here to help.
Polars is quickly becoming a go-to library for data scientists seeking faster and more efficient data processing capabilities. This Python library, built on a Rust foundation, offers exceptional performance for large datasets while maintaining an intuitive interface. This Statology Sprint brings together our most valuable Polars content to help you build proficiency with this powerful library.
Getting Started with Polars
Polars offers a compelling alternative to pandas with its lightning-fast performance and memory efficiency. To begin using Polars, you’ll first need to install it:
# Standard installation pip install polars # For processors without AVX support (pre-2011 CPUs, Apple Silicon) pip install polars-lts-cpu
If you’re transitioning from pandas, you can easily convert your existing DataFrames:
import pandas as pd
import polars as pl
# Convert pandas DataFrame to Polars
pandas_df = pd.DataFrame({'A': [1, 2, 3], 'B': ['a', 'b', 'c']})
polars_df = pl.from_pandas(pandas_df)
➡️ Learn more about converting from Pandas to Polars.
Core DataFrame Operations
Polars provides intuitive methods for creating and manipulating DataFrames. Here are some essential operations:
1. Adding New Columns
The with_columns() method allows you to add columns based on constants, calculations, or expressions:
df = pl.DataFrame({
"name": ["Alice", "Bob", "Charlie"],
"age": [25, 30, 35]
})
# Add a new column with a constant value
df_new = df.with_columns(pl.lit("USA").alias("country"))
# Add a calculated column
df_calc = df.with_columns((pl.col("age") * 2).alias("double_age"))
➡️ Learn more about adding columns with with_columns()
2. Selecting Specific Columns
Use select() to choose which columns to keep in your DataFrame:
# Select specific columns
result = df.select(['name', 'age'])
# Select with pattern matching
result = df.select([pl.col('^score_.*$')]) # All columns starting with "score_"
# Select all excluding specific columns
result = df.select(pl.col("*").exclude("age", "city"))
➡️ Learn more about selecting columns with select()
3. Filtering Rows
Filter rows based on conditions using the filter() method:
# Basic filtering
filtered_df = df.filter(pl.col('age') > 25)
# Multiple conditions
filtered_df = df.filter((pl.col('age') > 25) & (pl.col('city').is_in(['London', 'Paris'])))
# String operations
filtered_df = df.filter(pl.col('name').str.starts_with('A'))
➡️ Learn more about filtering rows with filter()
4. Sorting Data
Order your DataFrame by one or more columns using sort():
# Sort by a single column in ascending order
df_sorted = df.sort('age')
# Sort by multiple columns with different directions
df_sorted = df.sort(['department', 'salary'], descending=[False, True])
➡️ Learn more about sorting with sort()
5. Handling Missing Data
Deal with null values using the fill_null() method:
# Fill null values with a constant
df_filled = df.fill_null(0)
# Fill with type-appropriate constants
df_filled = df.with_columns([
pl.col("A").fill_null(0),
pl.col("B").fill_null("unknown")
])
# Use different fill strategies
df_forward = df.fill_null(strategy='forward') # Propagate last valid value
df_mean = df.with_columns(pl.col("value").fill_null(pl.col("value").mean()))
➡️ Learn more about handling missing data with fill_null()
Advanced DataFrame Operations
1. Concatenating DataFrames
Combine multiple DataFrames using the concat() function:
# Vertical concatenation (default) result = pl.concat([df1, df2]) # Horizontal concatenation result = pl.concat([df1, df2], how='horizontal') # Diagonal concatenation (union of columns) result = pl.concat([df1, df2], how='diagonal')
➡️ Learn more about concatenating DataFrames with concat()
2. Computing Statistics
Get a quick overview of your data with the describe() method:
# Basic usage stats = df.describe() # Custom percentiles custom_stats = df.describe(percentiles=[0.1, 0.5, 0.9]) # On specific columns length_stats = df.select(['sepal length', 'petal length']).describe()
➡️ Learn more about computing statistics with describe()
3. Performing Window Functions
Calculate metrics across related rows using window functions with over():
# Calculate cumulative sum within categories
df_with_cumsum = df.with_columns([
pl.col('sales').cum_sum().over(['category']).alias('cumulative_sales_by_category')
])
# Calculate moving averages
df_with_ma = df.with_columns([
pl.col('sales').rolling_mean(window_size=2).over(['category']).alias('2day_moving_avg')
])
➡️ Learn more about window functions with over()
4. Group Operations
Perform aggregations by groups using the group_by() method:
# Basic groupby with single aggregation
result = df.group_by('category').agg(pl.col('value').mean())
# Multiple aggregations
result = df.group_by('department').agg([
pl.col('salary').mean().alias('avg_salary'),
pl.col('salary').max().alias('max_salary'),
pl.col('years_exp').sum().alias('total_experience')
])
➡️ Learn more about group operations with group_by()
Working with LazyFrames
Polars’ LazyFrames defer computation until explicitly requested, enabling optimized query planning and better memory usage for complex operations.
# Convert DataFrame to LazyFrame
lazy_df = df.lazy()
# Define operations
query = (lazy_df
.filter(pl.col('age') > 25)
.select(['name', 'age', pl.col('salary').alias('annual_salary')])
)
# Execute the query
final_df = query.collect()
LazyFrames excel at complex data operations through a variety of specialized capabilities. They provide robust schema management and type handling to ensure data integrity across transformations. For sophisticated data combinations, LazyFrames offer optimized join operations that intelligently structure query execution for maximum efficiency. When troubleshooting or fine-tuning performance, the explain() method reveals query optimization details, allowing you to understand how Polars processes your data behind the scenes. For computationally intensive workflows, the cache()function strategically stores intermediate results, preventing redundant calculations and significantly improving performance with large datasets. These features make LazyFrames particularly valuable when working with complex data pipelines that require both performance and flexibility.
Working with Special Data Types
Temporal Data
Polars offers specialized functions for working with dates and times:
# Extract date components
df_extracted = df.select([
pl.col('date').dt.year().alias('year'),
pl.col('date').dt.month().alias('month'),
pl.col('time').dt.hour().alias('hour')
])
# Date arithmetic
df_date_math = df.select([
(pl.col('date') + pl.duration(days=7)).alias('date_plus_7_days')
])
# Timezone conversions
df_timezone = df.select([
pl.col('datetime').dt.replace_time_zone('UTC').alias('utc_time')
])
➡️ Learn more about temporal data in Polars
File I/O Operations
1. Reading CSV Files
You can efficiently load CSV files, even large ones, using read_csv():
# Basic reading
df = pl.read_csv('data.csv')
# Selective column reading
df = pl.read_csv('data.csv', columns=['MedInc', 'AveRooms'])
# Lazy reading for large files
lazy_df = pl.scan_csv('data.csv')
➡️ Learn more about reading CSV files with read_csv()
2. Writing to CSV Files
Save your DataFrames to CSV format using write_csv():
# Basic usage
df.write_csv("output.csv")
# Custom null value representation
df.write_csv("output_nulls.csv", null_value="N/A")
# Write a subset of columns
df.select(['Name', 'City']).write_csv("output_subset.csv")
➡️ Learn more about writing CSV files with write_csv()
Performance Considerations
Polars consistently outperforms pandas in benchmark tests:
| Operation | Speed Improvement |
|---|---|
| Filtering | ~4x faster |
| Aggregation | ~22x faster |
| GroupBy | ~8x faster |
| Sorting | ~3x faster |
| Feature Engineering | ~5.6x faster |
These performance gains become more significant with larger datasets, making Polars an excellent choice for data-intensive applications.
➡️ Learn more about performance benchmarks
Bringing It All Together: Wine Quality Analysis
Let’s combine several Polars techniques in a practical exercise using scikit-learn’s wine dataset.
import polars as pl
from sklearn.datasets import load_wine
import numpy as np
# Load the wine dataset
wine_data = load_wine()
feature_names = wine_data.feature_names
target_names = wine_data.target_names
# Create a Polars DataFrame
df = pl.DataFrame(
wine_data.data,
schema=[name.replace(' ', '_') for name in feature_names]
)
df = df.with_columns(pl.lit(wine_data.target).alias('wine_class'))
# Add class labels as strings for readability
wine_type_mapping = {i: name for i, name in enumerate(target_names)}
df = df.with_columns(
pl.col('wine_class').replace_strict(wine_type_mapping).alias('wine_type')
)
print(f"Wine dataset shape: {df.shape}")
print(df.head(3))
Wine dataset shape: (178, 15) shape: (3, 15) ┌─────────┬────────────┬──────┬──────────────┬───┬──────────────┬─────────┬────────────┬───────────┐ │ alcohol ┆ malic_acid ┆ ash ┆ alcalinity_o ┆ … ┆ od280/od315_ ┆ proline ┆ wine_class ┆ wine_type │ │ --- ┆ --- ┆ --- ┆ f_ash ┆ ┆ of_diluted_w ┆ --- ┆ --- ┆ --- │ │ f64 ┆ f64 ┆ f64 ┆ --- ┆ ┆ ines ┆ f64 ┆ i64 ┆ str │ │ ┆ ┆ ┆ f64 ┆ ┆ --- ┆ ┆ ┆ │ │ ┆ ┆ ┆ ┆ ┆ f64 ┆ ┆ ┆ │ ╞═════════╪════════════╪══════╪══════════════╪═══╪══════════════╪═════════╪════════════╪═══════════╡ │ 14.23 ┆ 1.71 ┆ 2.43 ┆ 15.6 ┆ … ┆ 3.92 ┆ 1065.0 ┆ 0 ┆ class_0 │ │ 13.2 ┆ 1.78 ┆ 2.14 ┆ 11.2 ┆ … ┆ 3.4 ┆ 1050.0 ┆ 0 ┆ class_0 │ │ 13.16 ┆ 2.36 ┆ 2.67 ┆ 18.6 ┆ … ┆ 3.17 ┆ 1185.0 ┆ 0 ┆ class_0 │ └─────────┴────────────┴──────┴──────────────┴───┴──────────────┴─────────┴────────────┴───────────┘
We’ve loaded the wine dataset and created a Polars DataFrame with 178 samples. The replace_strict() method efficiently converts numeric class IDs to readable string labels.
Step 1: Exploratory Analysis
# Get a statistical overview using describe()
stats = df.describe()
# Calculate class-specific statistics using group_by()
class_stats = df.group_by('wine_type').agg([
pl.col('alcohol').mean().alias('avg_alcohol'),
pl.col('flavanoids').mean().alias('avg_flavanoids'),
pl.col('color_intensity').mean().alias('avg_color')
])
print(class_stats)
shape: (3, 4) ┌───────────┬─────────────┬────────────────┬───────────┐ │ wine_type ┆ avg_alcohol ┆ avg_flavanoids ┆ avg_color │ │ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ f64 ┆ f64 ┆ f64 │ ╞═══════════╪═════════════╪════════════════╪═══════════╡ │ class_1 ┆ 12.278732 ┆ 2.080845 ┆ 3.08662 │ │ class_0 ┆ 13.744746 ┆ 2.982373 ┆ 5.528305 │ │ class_2 ┆ 13.15375 ┆ 0.781458 ┆ 7.39625 │ └───────────┴─────────────┴────────────────┴───────────┘
The group_by operation reveals clear chemical differences between wine classes. Class_0 wines have the highest alcohol and flavanoid content, while class_2 wines have the highest color intensity but lowest flavanoid levels. Class_1 wines show moderate values across all metrics.
Step 2: Feature Engineering with LazyFrames
# Convert to LazyFrame for optimized processing
lazy_df = df.lazy()
# Create new features and normalize key columns
wine_features = (lazy_df
.with_columns([
# Create alcohol-to-phenols ratio
(pl.col('alcohol') / pl.col('total_phenols')).alias('alcohol_phenol_ratio'),
# Normalize key columns using min-max scaling
((pl.col('alcohol') - pl.col('alcohol').min()) /
(pl.col('alcohol').max() - pl.col('alcohol').min()))
.alias('alcohol_normalized')
])
# Filter to focus on wines with higher alcohol content
.filter(pl.col('alcohol') > pl.col('alcohol').mean())
)
# Execute and inspect results
wine_results = wine_features.collect()
print(f"Filtered dataset shape: {wine_results.shape}")
Filtered dataset shape: (92, 17)
Using LazyFrames, we’ve created two new features and filtered the dataset in a single operation. The query was optimized behind the scenes, and we only collected the results when needed. The filter reduced our dataset from 178 to 92 samples, focusing on wines with above-average alcohol content.
Step 3: Identifying Distinctive Characteristics
# Find the most distinctive features for each wine class using window functions
distinctive_features = (df
.with_columns([
# Calculate z-scores for each feature within each class
((pl.col('flavanoids') - pl.col('flavanoids').mean().over('wine_type')) /
pl.col('flavanoids').std().over('wine_type')).alias('flavanoids_z'),
((pl.col('color_intensity') - pl.col('color_intensity').mean().over('wine_type')) /
pl.col('color_intensity').std().over('wine_type')).alias('color_z')
])
# Sort by absolute z-score to find most distinctive samples
.sort(pl.col('flavanoids_z').abs(), descending=True)
)
# Show the most distinctive wines
top_distinctive = distinctive_features.select(
['wine_type', 'alcohol', 'flavanoids', 'flavanoids_z', 'color_intensity', 'color_z']
).head(5)
print("Most distinctive wines by flavanoid content:")
print(top_distinctive)
Most distinctive wines by flavanoid content: shape: (5, 6) ┌───────────┬─────────┬────────────┬──────────────┬─────────────────┬───────────┐ │ wine_type ┆ alcohol ┆ flavanoids ┆ flavanoids_z ┆ color_intensity ┆ color_z │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ f64 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═══════════╪═════════╪════════════╪══════════════╪═════════════════╪═══════════╡ │ class_1 ┆ 11.56 ┆ 5.08 ┆ 4.249896 ┆ 6.0 ┆ 3.149841 │ │ class_2 ┆ 13.5 ┆ 1.57 ┆ 2.686646 ┆ 8.6 ┆ 0.520891 │ │ class_0 ┆ 14.19 ┆ 3.93 ┆ 2.384006 ┆ 8.7 ┆ 2.560766 │ │ class_1 ┆ 12.37 ┆ 3.75 ┆ 2.365245 ┆ 4.5 ┆ 1.528096 │ │ class_1 ┆ 12.37 ┆ 0.57 ┆ -2.140915 ┆ 1.95 ┆ -1.228872 │ └───────────┴─────────┴────────────┴──────────────┴─────────────────┴───────────┘
Using window functions with over(), we’ve calculated z-scores for flavanoids and color intensity within each wine class. This reveals the most unusual wines in the dataset. The top result is a class_1 wine with flavanoid content 4.25 standard deviations above its class average. The last entry shows the opposite — a class_1 wine with unusually low flavanoid levels.
Step 4: Export Analysis Results
# Create a summary report with key characteristics by wine type
summary = (df
.group_by('wine_type')
.agg([
pl.len().alias('sample_count'),
pl.col('alcohol').mean().round(2).alias('avg_alcohol'),
pl.col('malic_acid').mean().round(2).alias('avg_malic_acid'),
pl.col('flavanoids').mean().round(2).alias('avg_flavanoids'),
pl.col('color_intensity').mean().round(2).alias('avg_color')
])
)
# Save results to CSV
summary.write_csv("wine_analysis_summary.csv")
print("\nAnalysis summary:")
print(summary)
Analysis summary: shape: (3, 6) ┌───────────┬──────────────┬─────────────┬────────────────┬────────────────┬───────────┐ │ wine_type ┆ sample_count ┆ avg_alcohol ┆ avg_malic_acid ┆ avg_flavanoids ┆ avg_color │ │ --- ┆ --- ┆ --- ┆ --- ┆ --- ┆ --- │ │ str ┆ u32 ┆ f64 ┆ f64 ┆ f64 ┆ f64 │ ╞═══════════╪══════════════╪═════════════╪════════════════╪════════════════╪═══════════╡ │ class_2 ┆ 48 ┆ 13.15 ┆ 3.33 ┆ 0.78 ┆ 7.4 │ │ class_0 ┆ 59 ┆ 13.74 ┆ 2.01 ┆ 2.98 ┆ 5.53 │ │ class_1 ┆ 71 ┆ 12.28 ┆ 1.93 ┆ 2.08 ┆ 3.09 │ └───────────┴──────────────┴─────────────┴────────────────┴────────────────┴───────────┘
Our final summary table combines multiple aggregation functions in a single group_by operation, creating a comprehensive overview of the dataset. The table reveals that class_1 is the most common wine type (71 samples), while class_2 has the highest malic acid content and color intensity but the lowest flavanoid levels. This summary was automatically exported to CSV, ready to be shared or used in other applications. This capstone exercise demonstrates Polars’ ability to efficiently handle:
- Statistical analysis with describe() and group_by()
- Feature engineering with LazyFrames for optimized execution
- Window functions for comparative analysis within groups
- Clean data export for sharing results
By combining these techniques, you can quickly extract meaningful insights from your datasets while maintaining excellent performance. As you continue your Polars journey, consider exploring these additional topics:
- Integrating Polars with machine learning workflows
- Using expressions for complex data transformations
- Implementing custom user-defined functions (UDFs)
- Combining Polars with visualization libraries
- Advanced time series analysis
The performance benefits and intuitive API make Polars worth investing time to learn, especially if you work with large datasets or require efficient data processing pipelines. By exploring the techniques covered in this Statology Sprint, you’ll be well on your way to leveraging Polars for fast, memory-efficient data analysis in Python.
