Image by Editor

Oftentimes, building a strong analysis solution or predictive model entails incorporating relevant data that belongs to a mix of types — not just numerical and categorical data, but also temporal, ordinal, or even unstructured data such as text and images. Unsurprisingly, dealing with datasets containing mixed data types is one of the most frequent hurdles a data scientist may have to face.

The good news is that there are a variety of strategies for translating or unifying these disparate formats into a common representation that algorithms can comprehend, all without incurring significant information loss. This article uncovers some of the most notable methodological approaches to dealing with mixed data.

The Universal Translator: Feature Encoding

Feature encoding primarily encompasses strategies for converting categorical data into a numerical format. Depending on the nature of your categorical variables, you may want to apply one of these strategies to each variable:

• One-hot encoding: An essential concept in data preprocessing and feature transformation for machine learning. It consists of creating multiple new binary features: one per category. Use this for a categorical variable with few unique values, i.e. low cardinality.
• Target encoding: Suppose you have high-cardinality categorical variables, such as the country of origin of a customer in a dataset associated with a worldwide company. In these situations, replacing each category with the mean of the target variable for that category might be an effective approach, although it normally requires cross-validation afterward to prevent potential issues like data leakage.
• Ordinal encoding: Do categories have a semantic sense of order? Can they be logically ranked? Then look no further, and apply a simple ordinal encoding, e.g. “small”=1, “medium”=2, “large”=3, and so on.

Specialized Distance Metrics and Strategies

When we have mixed data types, standard distance metrics like Manhattan and Euclidean distance are likely to fall short. Luckily, some strategies directly tackle this issue, not only for computing distances at the instance level, but also for applying distance-based algorithms like clustering:

• Gower’s distance: The “gold standard” for measuring distances between mixed data points. It simply consists of computing a 0-1 distance feature-wise. This process is applied differently depending on the specific feature’s type. Then, we calculate a weighted average of feature-level distances to obtain the overall distance. This article helps you understand this distance metric in detail.
• K-prototypes: In need of clustering mixed data? K-prototypes is a variant of k-means clustering that has been specifically redesigned to handle mixed data. While it utilizes Euclidean distance for numerical attributes, it also applies simple matching rules for categorical ones, thereby avoiding the need to discard them.

Algorithmic Adaptation

In machine learning and advanced statistical analysis, certain models are “type-agnostic.” In other words, they do not need arduous data preprocessing to deal with mixed data before using it. These include:

• Tree-based models and ensembles: Random forests, GradientBoosting, and XGBoost are some of the most popular terms in machine learning jargon. These are versatile, tree-based models that can handle mixed data easily. Numbers are turned into thresholds, and categories are turned into membership-based rules. CatBoost is a particular type of ensemble designed to handle categorical features elegantly.
• Factor analysis of mixed data (FAMD): A variant of the popular PCA method for dimensionality reduction that combines basic PCA principles with multiple correspondence analysis to incorporate qualitative data.

Normalization and Transformation

Be aware that when there are mixed data types in a dataset, the scales of numerical data can exacerbate the risk of overshadowing categorical signals. Scaling techniques like standardization and min-max scaling become particularly necessary. Likewise, trying to make numerical feature distributions more normal-like, or Gaussian, can help detect relationships with categorical variables more easily. Approaches like Yeo-Johnson and Box-Cox transformations are designed for this purpose.

How About Text, Images, and Temporal Data?

When we have text, images, or temporal data that may contain valuable information for downstream analysis, we need a way to integrate them with other data types expressed in tabular format, like the numerical and categorical data seen thus far.

For text data, solutions to turn raw text into numerical features and representations vary, from classical approaches like TF-IDF to the modern standard: embeddings. Images can be passed to a pre-trained convolutional neural network model like ResNet to obtain an image embedding: a fixed-size vector that encodes the most important image features numerically. As for temporal data, there are plenty of strategies to extract purely numerical attributes, e.g. deconstructing timestamps into components like hour, day of week, weekend vs. weekday, etc., and calculating rolling aggregations in time series.

Summary

This article took a tour through a variety of strategies and methodologies for handling datasets containing a variety of data types. While most strategies mainly focus on data that combines numerical and categorical features, it is not uncommon in modern datasets and analysis scenarios to have other complex data types in the scene, such as text, images, and time series. In upcoming articles, we will learn in more detail about some of the strategies outlined here.