Combining multiple DataFrames is a common data science operation and Polars provides an efficient solution with its concat() function. Here are some of its main features: 

1. By default, concat() combines DataFrames vertically (along the rows).
2. You can use the how parameter to specify different concatenation methods.
3. The function can handle DataFrames with different column names and data types.

In this guide, we’ll explore different strategies for concatenation using Polars’ concat() function, with practical examples for each approach.

Example 1: Basic Vertical Concatenation

We will start by creating two DataFrames and concatenate them vertically.

import polars as pl
# Create two sample DataFrames
df1 = pl.DataFrame({
    'A': [1, 2, 3],
    'B': ['a', 'b', 'c']
})
df2 = pl.DataFrame({
    'A': [4, 5, 6],
    'B': ['d', 'e', 'f']
})
# Concatenate vertically
result = pl.concat([df1, df2])
print(result)

Output:

shape: (6, 2)
┌─────┬─────┐
│ A   ┆ B   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ a   │
│ 2   ┆ b   │
│ 3   ┆ c   │
│ 4   ┆ d   │
│ 5   ┆ e   │
│ 6   ┆ f   │
└─────┴─────┘

Here we created two DataFrames with the same structure and used pl.concat() to combine them vertically. The resulting DataFrame contains all rows from both input DataFrames.

Example 2: Horizontal Concatenation

Now, let’s look at how to concatenate DataFrames horizontally using the how=’horizontal’ parameter.

import polars as pl
# Create two sample DataFrames with different columns
df1 = pl.DataFrame({
    'A': [1, 2, 3],
    'B': ['a', 'b', 'c']
})
df2 = pl.DataFrame({
    'C': [True, False, True],
    'D': [4.5, 5.5, 6.5]
})
# Concatenate horizontally
result = pl.concat([df1, df2], how='horizontal')
print(result)

Output:

shape: (3, 4)
┌─────┬─────┬───────┬─────┐
│ A   ┆ B   ┆ C     ┆ D   │
│ --- ┆ --- ┆ ---   ┆ --- │
│ i64 ┆ str ┆ bool  ┆ f64 │
╞═════╪═════╪═══════╪═════╡
│ 1   ┆ a   ┆ true  ┆ 4.5 │
│ 2   ┆ b   ┆ false ┆ 5.5 │
│ 3   ┆ c   ┆ true  ┆ 6.5 │
└─────┴─────┴───────┴─────┘

In this example, we concatenated two DataFrames with different column names horizontally. The resulting DataFrame contains all columns from both input DataFrames.

Example 3: Diagonal Concatenation

The diagonal strategy combines the DataFrames by creating a union of their columns. Any column values that are missing in the DataFrames will be filled with null values.

import polars as pl
# Create DataFrames with different structures
df1 = pl.DataFrame({
    'A': [1, 2, 3],
    'B': ['a', 'b', 'c']
})
df2 = pl.DataFrame({
    'B': ['d', 'e'],
    'C': [True, False]
})
# Concatenate diagonally
result = pl.concat([df1, df2], how='diagonal')
print("DataFrame 1:")
print(df1)
print("\nDataFrame 2:")
print(df2)
print("\nResult:")
print(result)

Output:

DataFrame 1:
shape: (3, 2)
┌─────┬─────┐
│ A   ┆ B   │
│ --- ┆ --- │
│ i64 ┆ str │
╞═════╪═════╡
│ 1   ┆ a   │
│ 2   ┆ b   │
│ 3   ┆ c   │
└─────┴─────┘

DataFrame 2:
shape: (2, 2)
┌─────┬───────┐
│ B   ┆ C     │
│ --- ┆ ---   │
│ str ┆ bool  │
╞═════╪═══════╡
│ d   ┆ true  │
│ e   ┆ false │
└─────┴───────┘

Result:
shape: (5, 3)
┌──────┬─────┬───────┐
│ A    ┆ B   ┆ C     │
│ ---  ┆ --- ┆ ---   │
│ i64  ┆ str ┆ bool  │
╞══════╪═════╪═══════╡
│ 1    ┆ a   ┆ null  │
│ 2    ┆ b   ┆ null  │
│ 3    ┆ c   ┆ null  │
│ null ┆ d   ┆ true  │
│ null ┆ e   ┆ false │
└──────┴─────┴───────┘

In this example, we concatenated two DataFrames with different column structures using how=’diagonal’. Polars creates a union of all columns and fills missing values with null for columns that were not present in both DataFrames.

Example 4: Align Concatenation

The align strategy in concat() combines frames horizontally, automatically determining the common key columns and aligning rows. This behavior is patterned after a full outer join but does not handle column-name collisions.

import polars as pl
# Create two DataFrames with some overlapping and some unique data
df1 = pl.DataFrame({
    'ID': [1, 2, 3, 4],
    'Name': ['Alice', 'Bob', 'Charlie', 'David'],
    'Age': [25, 30, 35, 40]
})
df2 = pl.DataFrame({
    'ID': [3, 4, 5, 6],
    'Name': ['Charlie', 'David', 'Eve', 'Frank'],
    'City': ['New York', 'Los Angeles', 'Chicago', 'Houston']
})
# Concatenate using align strategy
result = pl.concat([df1, df2], how='align')
print("DataFrame 1:")
print(df1)
print("\nDataFrame 2:")
print(df2)
print("\nResult:")
print(result)

Output:

DataFrame 1:
shape: (4, 3)
┌─────┬─────────┬─────┐
│ ID  ┆ Name    ┆ Age │
│ --- ┆ ---     ┆ --- │
│ i64 ┆ str     ┆ i64 │
╞═════╪═════════╪═════╡
│ 1   ┆ Alice   ┆ 25  │
│ 2   ┆ Bob     ┆ 30  │
│ 3   ┆ Charlie ┆ 35  │
│ 4   ┆ David   ┆ 40  │
└─────┴─────────┴─────┘

DataFrame 2:
shape: (4, 3)
┌─────┬─────────┬─────────────┐
│ ID  ┆ Name    ┆ City        │
│ --- ┆ ---     ┆ ---         │
│ i64 ┆ str     ┆ str         │
╞═════╪═════════╪═════════════╡
│ 3   ┆ Charlie ┆ New York    │
│ 4   ┆ David   ┆ Los Angeles │
│ 5   ┆ Eve     ┆ Chicago     │
│ 6   ┆ Frank   ┆ Houston     │
└─────┴─────────┴─────────────┘

Result:
shape: (6, 4)
┌─────┬─────────┬──────┬─────────────┐
│ ID  ┆ Name    ┆ Age  ┆ City        │
│ --- ┆ ---     ┆ ---  ┆ ---         │
│ i64 ┆ str     ┆ i64  ┆ str         │
╞═════╪═════════╪══════╪═════════════╡
│ 1   ┆ Alice   ┆ 25   ┆ null        │
│ 2   ┆ Bob     ┆ 30   ┆ null        │
│ 3   ┆ Charlie ┆ 35   ┆ New York    │
│ 4   ┆ David   ┆ 40   ┆ Los Angeles │
│ 5   ┆ Eve     ┆ null ┆ Chicago     │
│ 6   ┆ Frank   ┆ null ┆ Houston     │
└─────┴─────────┴──────┴─────────────┘

In this example:

1. We created two DataFrames (df1 and df2) with some overlapping IDs and Names, but different column structures.
2. We used pl.concat() with how=’align’.
3. The resulting DataFrame:
   • Combines all unique rows from both DataFrames.
   • Automatically determines common key columns (‘ID’ and ‘Name’ in this case) for alignment.
   • Combines all columns from both DataFrames.
   • Fills in null values for missing data where rows don’t match across DataFrames.

Few points to note about the align strategy:

• Polars automatically determines the common key columns between the DataFrames for alignment.
• The behavior is patterned after a full outer join, but it does not handle column-name collisions.
• This strategy is particularly useful when you have DataFrames with partially overlapping structure but want to combine all unique information without explicitly specifying key columns.
• If you need more control over the alignment process, you might want to consider using the polars.align_frames() method instead. This function allows for more fine-grained control over the alignment process, including the ability to specify key columns using the on parameter. 

Conclusion

The concat() function in Polars is a flexible and efficient tool for combining multiple DataFrames. It offers four main strategies:

1. Vertical concatenation (default)
2. Horizontal concatenation
3. Diagonal concatenation
4. Align concatenation

Each strategy serves different purposes and can be useful depending on your specific data manipulation needs. Remember to consider the structure of your DataFrames and the relationships between them when choosing the appropriate concatenation method. With practice, you’ll find that the Polars’ concat() function becomes an indispensable part of your data science workflow.