Filtering rows in a Polars DataFrame using the filter() method allows you to select specific data based on certain conditions. This technique is useful for data manipulation and analysis in Python, enabling you to work with subsets of your data that meet specific criteria. The filter() method returns a new DataFrame containing only the rows that satisfy the specified conditions, helping you focus on the data that’s most relevant to your analysis.
When you filter rows, you’re essentially specifying which rows you want to keep based on their values. This can be done using various expressions and constraints. Filtering is particularly useful when you need to answer questions that involve phrases such as “is at least,” “is equal to,” or “is greater than.”
Let’s work through some examples to see how we can use filter() in different scenarios.
Example 1: Basic Filtering
First, let’s create a sample DataFrame and perform a simple filter operation.
import polars as pl
# Create a sample DataFrame
df = pl.DataFrame({
'name': ['Alice', 'Bob', 'Charlie', 'David', 'Eve'],
'age': [25, 30, 35, 28, 22],
'city': ['New York', 'London', 'Paris', 'Tokyo', 'Sydney']
})
# Print the original DataFrame
print(df)
# Filter rows where age is greater than 25
filtered_df = df.filter(pl.col('age') > 25)
print(filtered_df)
Output:
shape: (5, 3) ┌─────────┬─────┬──────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪══════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ │ David ┆ 28 ┆ Tokyo │ │ Eve ┆ 22 ┆ Sydney │ └─────────┴─────┴──────────┘ shape: (3, 3) ┌─────────┬─────┬────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪════════╡ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ │ David ┆ 28 ┆ Tokyo │ └─────────┴─────┴────────┘
In this example, we filter the DataFrame to include only rows where the ‘age’ is greater than 25. This demonstrates how we can use filtering to answer questions like “Which individuals are older than 25 years?”
Example 2: Multiple Conditions
You can combine multiple conditions using logical operators like & (and) and | (or). This allows for more complex filtering scenarios.
# Print the original DataFrame
print(df)
# Filter rows where age is greater than 25 and city is either London or Paris
filtered_df = df.filter((pl.col('age') > 25) & (pl.col('city').is_in(['London', 'Paris'])))
print(filtered_df)
Output:
shape: (5, 3) ┌─────────┬─────┬──────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪══════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ │ David ┆ 28 ┆ Tokyo │ │ Eve ┆ 22 ┆ Sydney │ └─────────┴─────┴──────────┘ shape: (2, 3) ┌─────────┬─────┬────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪════════╡ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ └─────────┴─────┴────────┘
This filter selects rows where the age is greater than 25 and the city is either London or Paris. It showcases how we can use multiple constraints to narrow down our data selection, answering questions like “Who are the individuals over 25 years old living in either London or Paris?”
Example 3: String Operations
Polars allows you to use string operations in your filter conditions, which is particularly useful when working with text data. Let’s see how we can filter rows based on string patterns.
# Print the original DataFrame
print(df)
# Filter rows where the name starts with 'A' or 'B'
filtered_df = df.filter(pl.col('name').str.starts_with('A') | pl.col('name').str.starts_with('B'))
print(filtered_df)
Output:
shape: (5, 3) ┌─────────┬─────┬──────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪══════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ │ David ┆ 28 ┆ Tokyo │ │ Eve ┆ 22 ┆ Sydney │ └─────────┴─────┴──────────┘ shape: (2, 3) ┌───────┬─────┬──────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═══════╪═════╪══════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ London │ └───────┴─────┴──────────┘
In this example, we filter the DataFrame to include only rows where the ‘name’ starts with either ‘A’ or ‘B’. We use the str.starts_with() method for each prefix and combine the conditions with the OR operator (|).
This demonstrates how to filter rows based on string patterns, which is excellent for working with textual data. It allows us to answer questions like “Which individuals have names beginning with certain letters?”
Example 4: Chaining Filters
Polars allows you to chain multiple filter() operations, which can be an efficient way to apply sequential filters to your data. Let’s see how this works:
# Print the original DataFrame
print(df)
# Chain filters: first filter age > 25, then filter city == 'London'
result = df.filter(pl.col('age') > 25).filter(pl.col('city') == 'London')
print(result)
Output:
shape: (5, 3) ┌─────────┬─────┬──────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞═════════╪═════╪══════════╡ │ Alice ┆ 25 ┆ New York │ │ Bob ┆ 30 ┆ London │ │ Charlie ┆ 35 ┆ Paris │ │ David ┆ 28 ┆ Tokyo │ │ Eve ┆ 22 ┆ Sydney │ └─────────┴─────┴──────────┘ shape: (1, 3) ┌──────┬─────┬────────┐ │ name ┆ age ┆ city │ │ --- ┆ --- ┆ --- │ │ str ┆ i64 ┆ str │ ╞══════╪═════╪════════╡ │ Bob ┆ 30 ┆ London │ └──────┴─────┴────────┘
In this example, we chain two filter operations:
- First, we filter for rows where the age is greater than 25.
- Then, from that result, we filter for rows where the city is ‘London’.
This chaining technique allows us to apply multiple filters sequentially, refining our data step by step. It’s particularly useful when you need to apply a series of conditions to your data, each building upon the results of the previous filter.
In this case, our chained filters answer the question: “Who is over 25 years old and lives in London?” The result shows that only Bob meets both criteria.
Chaining filters can be more readable and maintainable than combining multiple conditions in a single filter, especially when dealing with complex filtering logic. It also allows you to easily add or remove filtering steps as your analysis requirements change.
By applying these filtering techniques in Polars, you’ll be able to efficiently select and analyze subsets of your data. Filtering allows you to focus on the most relevant data for your analysis, helping you extract valuable insights from your datasets. As you practice with these techniques, you’ll become more proficient in your data analysis tasks, enabling you to handle complex data processing challenges with ease.
