Missing data is a frequent challenge in data analysis, often requiring thoughtful strategies to address it effectively. In Polars, null values are used to represent missing data, which can arise from incomplete information, errors, or intentional omissions. The fill_null() function offers a versatile solution by allowing you to replace null values in a DataFrame using constants or various fill strategies.
In this post, we’ll explore how to use fill_nullI() effectively to handle missing data in Polars, starting with basic usage and progressing to more advanced techniques.
Example 1A: Basic Usage of fill_null()
Let’s start with a simple example of using fill_null() to replace missing values with a constant:
import polars as pl
# Create a sample DataFrame with missing values
df = pl.DataFrame({
'A': [1, None, 3, None, 5],
'B': ['a', 'b', None, 'd', None]
})
# Fill null values with a constant
df_filled = df.fill_null(0)
print(df_filled)
This will output:
shape: (5, 2) ┌─────┬──────┐ │ A ┆ B │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪══════╡ │ 1 ┆ a │ │ 0 ┆ b │ │ 3 ┆ null │ │ 0 ┆ d │ │ 5 ┆ null │ └─────┴──────┘
You might notice that the null values in column B are not replaced. This is because Polars handles fill_null() in a type-safe manner. When we use fill_null(0), Polars attempts to fill null values with the provided value (0 in this case), but it only does so for columns where the type is compatible. For column A (integers), filling with 0 works as expected. However, for column B (strings), Polars cannot fill string nulls with an integer 0, so the null values remain unchanged.
Example 1B: Type-Appropriate fill_null()
To properly handle different column types, we need to apply fill_null() to each column separately with its appropriate fill value:
import polars as pl
# Create a sample DataFrame with missing values
df = pl.DataFrame({
'A': [1, None, 3, None, 5],
'B': ['a', 'b', None, 'd', None]
})
# Fill null values with type-appropriate constants for each column
df_filled = df.with_columns([
pl.col("A").fill_null(0),
pl.col("B").fill_null("unknown")
])
print(df_filled)
This will output:
shape: (5, 2) ┌─────┬─────────┐ │ A ┆ B │ │ --- ┆ --- │ │ i64 ┆ str │ ╞═════╪═════════╡ │ 1 ┆ a │ │ 0 ┆ b │ │ 3 ┆ unknown │ │ 0 ┆ d │ │ 5 ┆ unknown │ └─────┴─────────┘
In this example, we use the with_columns() method along with pl.col() to apply fill_null() to each column separately. For the integer column ‘A’, we fill with 0, and for the string column ‘B’, we fill with ‘unknown’. This approach ensures that all null values are replaced with appropriate values for their respective column types.
It’s important to note that while some DataFrame libraries allow passing a dictionary to fill null values for specific columns, Polars requires us to explicitly apply the fill operation to each column. This approach, while slightly more verbose, gives us fine-grained control over how we handle missing data in each column.
Fill Strategies in fill_null()
Polars offers various strategies for filling null values beyond simple constants. When using fill_null(), you can specify a fill strategy that determines how missing values are imputed.
Available fill strategies in Polars include:
- None: Do not fill missing values.
- ‘forward’: Fill with the previous non-null value.
- ‘backward’: Fill with the next non-null value.
- ‘min’: Fill with the minimum value of the column.
- ‘max’: Fill with the maximum value of the column.
- ‘mean’: Fill with the mean of the column. Note that this mean is cast to the data type of the column, which in the case of an integer means the part behind the decimal point is truncated.
- ‘zero’: Fill with 0.
- ‘one’: Fill with 1.
These strategies provide flexible options for handling missing data based on the nature of your dataset and analysis requirements.
Example 2: Using Forward and Backward Fill Strategies
Let’s demonstrate the use of ‘forward’ and ‘backward’ fill strategies:
import polars as pl
# Create a sample DataFrame with a numeric series containing null values
df = pl.DataFrame({
'A': [1, None, 3, None, 5, None, 7]
})
# Apply forward fill strategy
df_forward = df.fill_null(strategy='forward')
# Apply backward fill strategy
df_backward = df.fill_null(strategy='backward')
print("Original DataFrame:")
print(df)
print("\nForward Fill:")
print(df_forward)
print("\nBackward Fill:")
print(df_backward)
This will output:
Original DataFrame: shape: (7, 1) ┌──────┐ │ A │ │ --- │ │ i64 │ ╞══════╡ │ 1 │ │ null │ │ 3 │ │ null │ │ 5 │ │ null │ │ 7 │ └──────┘ Forward Fill: shape: (7, 1) ┌─────┐ │ A │ │ --- │ │ i64 │ ╞═════╡ │ 1 │ │ 1 │ │ 3 │ │ 3 │ │ 5 │ │ 5 │ │ 7 │ └─────┘ Backward Fill: shape: (7, 1) ┌─────┐ │ A │ │ --- │ │ i64 │ ╞═════╡ │ 1 │ │ 3 │ │ 3 │ │ 5 │ │ 5 │ │ 7 │ │ 7 │ └─────┘
In this example:
- We create a DataFrame with a single column ‘A’ containing some null values.
- We apply the ‘forward’ fill strategy, which propagates the last non-null value forward to fill null values.
- We apply the ‘backward’ fill strategy, which fills null values with the next non-null value in the series.
These strategies are particularly useful when you want to fill missing values based on the surrounding data, rather than a fixed value. The forward fill is often used in time series data where you want to carry forward the last known value, while backward fill can be useful when you want to use future known values to fill past missing data.
Example 3: Filling Null Values with Column Mean Using Expressions
One powerful way to use fill_null() is by passing an expression that calculates a value based on the existing data. Here’s an example that fills null values with the mean of the column:
import polars as pl
# Create a sample DataFrame with missing values
df = pl.DataFrame({
'value': [10, None, 30, None, 50, 60, None]
})
# Fill null values with the mean of the column
df_filled = df.with_columns(
pl.col("value")
.fill_null(pl.col("value").mean())
.alias("value_filled_with_mean")
)
print("Original DataFrame:")
print(df)
print("\nDataFrame with nulls filled by mean:")
print(df_filled)
This will output:
Original DataFrame: shape: (7, 1) ┌───────┐ │ value │ │ --- │ │ i64 │ ╞═══════╡ │ 10 │ │ null │ │ 30 │ │ null │ │ 50 │ │ 60 │ │ null │ └───────┘ DataFrame with nulls filled by mean: shape: (7, 2) ┌───────┬────────────────────────┐ │ value ┆ value_filled_with_mean │ │ --- ┆ --- │ │ i64 ┆ f64 │ ╞═══════╪════════════════════════╡ │ 10 ┆ 10.0 │ │ null ┆ 37.5 │ │ 30 ┆ 30.0 │ │ null ┆ 37.5 │ │ 50 ┆ 50.0 │ │ 60 ┆ 60.0 │ │ null ┆ 37.5 │ └───────┴────────────────────────┘
In this example:
- We create a DataFrame with a single column ‘value’ containing some null values.
- We use with_columns() to create a new column while keeping the original.
- Inside with_columns(), we use fill_null() with an expression pl.col(“value”).mean().
- This expression calculates the mean of the ‘value’ column and uses it to fill the null values.
- We use alias() to name the new column ‘value_filled_with_mean’.
The result is a new column where all null values have been replaced with the mean of the original column (37.5 in this case).
This approach is particularly useful when you want to fill null values based on some calculation from the existing data. You could modify the expression to use other aggregations like median, mode, or even more complex calculations involving multiple columns.
Conclusion
Handling missing data is an important skill in data science and machine learning. Polars’ fill_null() method provides powerful and flexible ways to address this common challenge. By practicing these functions and understanding how they interact with different data types, you can ensure your data is clean and ready for analysis or model training.
Remember to consider the nature of your data and the implications of your chosen fill strategy. While filling missing values can be necessary, it’s important to understand how different approaches might affect your subsequent analyses or model performance.
