Image by Editor | ChatGPT
Introduction
Handling a time index correctly is crucial for feature engineering with time-based data. A time index is a column that records when each event happened, ensuring that features are calculated in the correct temporal order. Featuretools simplifies this process by using the time index to respect event sequences and prevent data leakage. For example, in a customer transaction dataset, the timestamp of each transaction is critical for building features that accurately reflect a customer’s history.
This article will demonstrate how to use the make_temporal_cutoffs function to create time-aware features, which improves model accuracy by ensuring calculations only use valid, past data.
Setting Up the Data
We will use a sample customer transactions dataset from Featuretools. It contains tables like customers, transactions, and sessions. Each transaction has a timestamp, which we will use as the time index.
import featuretools as ft # Load a mock dataset with transactions over time es = ft.demo.load_mock_customer(return_entityset=True) # Display the entity set print(es)
Entityset: transactions
DataFrames:
transactions [Rows: 500, Columns: 6]
products [Rows: 5, Columns: 3]
sessions [Rows: 35, Columns: 5]
customers [Rows: 5, Columns: 5]
Relationships:
transactions.product_id -> products.product_id
transactions.session_id -> sessions.session_id
sessions.customer_id -> customers.customer_id
Defining a Time Index
Next, we will update the transactions dataframe in our entity set to explicitly define the time index. While the demo loader often handles this, explicitly setting it ensures that feature calculations respect the chronological order of events. We use replace_dataframe to update the existing entity.
# Retrieve the transactions dataframe
transactions_df = es["transactions"].df
# Update the transactions dataframe definition to set a time index
es.replace_dataframe(
dataframe_name="transactions",
df=transactions_df,
index="transaction_id",
time_index="transaction_time"
)
- dataframe_name=”transactions”: Specifies the name of the dataframe in the entity set.
- df=transactions_df: The pandas dataframe containing the transaction data.
- index=”transaction_id”: The column that uniquely identifies each transaction.
- time_index=”transaction_time”: The column to be used as the time index.
Generating Time-Aware Features
Now that the time index is set, we can run Deep Feature Synthesis (DFS). This process will automatically generate features while respecting the temporal order defined by the time index.
# Run DFS with time-aware features
feature_matrix, feature_defs = ft.dfs(
entityset=es,
target_dataframe_name="customers",
agg_primitives=["sum", "mean", "count"],
trans_primitives=["month", "day"],
max_depth=2
)
# Display the feature matrix
print(feature_matrix.head())
zip_code COUNT(sessions) COUNT(transactions) \
customer_id
5 60091 6 79
4 60091 8 109
1 60091 8 126
3 13244 6 93
2 13244 7 93
MEAN(transactions.amount) SUM(transactions.amount) \
customer_id
5 80.375443 6349.66
4 80.070459 8727.68
1 71.631905 9025.62
3 67.060430 6236.62
2 77.422366 7200.28
DAY(birthday) DAY(join_date) MONTH(birthday) MONTH(join_date) \
customer_id
5 28 17 7 7
4 15 8 8 4
1 18 17 7 4
3 21 13 11 8
2 18 15 8 4
MEAN(sessions.COUNT(transactions)) \
customer_id
5 13.166667
4 13.625000
1 15.750000
3 15.500000
2 13.285714
MEAN(sessions.MEAN(transactions.amount)) \
customer_id
5 78.705187
4 81.207189
1 72.774140
3 67.539577
2 78.415122
MEAN(sessions.SUM(transactions.amount)) \
customer_id
5 1058.276667
4 1090.960000
1 1128.202500
3 1039.436667
2 1028.611429
SUM(sessions.MEAN(transactions.amount))
customer_id
5 472.231119
4 649.657515
1 582.193117
3 405.237462
2 548.905851
Using make_temporal_cutoffs
Featuretools provides the make_temporal_cutoffs function to generate a cutoff time dataframe. Instead of using a single cutoff time for all data, this function creates multiple cutoff points for each instance (e.g., each customer), which is essential for building time-series prediction models.
# Generate temporal cutoffs based on the transaction times
cutoff_times = ft.make_temporal_cutoffs(
es["customers"]["customer_id"],
es["transactions"]["transaction_time"],
window_size="1h",
num_windows=2
)
# Display the generated temporal cutoffs
print(cutoff_times)
time instance_id
0 2013-12-31 23:00:00 5
1 2014-01-01 00:00:00 5
2 2013-12-31 23:01:05 4
3 2014-01-01 00:01:05 4
4 2013-12-31 23:02:10 1
5 2014-01-01 00:02:10 1
6 2013-12-31 23:03:15 3
7 2014-01-01 00:03:15 3
8 2013-12-31 23:04:20 2
9 2014-01-01 00:04:20 2
- window_size=”1h”: This defines the amount of time between the last event in our data and the first generated cutoff time. Since the step parameter is not specified, it defaults to the window_size, meaning it also sets the interval between consecutive cutoffs at one hour.
- num_windows=2: This specifies how many cutoff windows to generate. For each customer instance, two distinct cutoff times will be created.
Conclusion
Setting a time index is fundamental to ensuring that features are generated based on the correct chronological order of events. This prevents data leakage — a common mistake where future data is used to calculate features for past events. By using a time index, we create features that accurately reflect historical data. Furthermore, using make_temporal_cutoffs allows us to generate feature vectors for specific time windows for each customer. This technique improves a model’s predictive accuracy by ensuring all calculations are historically valid and relevant to the point in time being predicted.
