Image by Author | Ideogram
In data science processes and building machine learning models, it is very common to resort to data sampling methods for a variety of reasons: from reducing computational load when working with a very large dataset to efficiently evaluating model performance without incurring excessive computing resource consumption. However, in most scenarios, sampling should not be done blindly — let alone relying only on purely random sampling methods — as problems may arise, such as unbalanced representation in the resulting sample, biased outcomes, and overlooked categories or subgroups that exist in the original data (particularly minorities).
To help you choose the right sampling method for your data and problem needs, this article provides a guide, centered around a decision tree, to navigate you toward the most appropriate approach based on use case, randomness requirements, and data characteristics.
Guide to Choose the Right Sampling Method
Our guide is centered on the following decision tree diagram: starting from the node on the left-hand side, answering a series of questions will help you navigate towards the most suitable data sampling approach.
Choosing the Right Sampling Approach (click to enlarge)
The eight sampling methods outlined are among the most popular and widely utilized ones in real-world scenarios. Below, we briefly discuss them, along with the key questions leading to using one method or another.
Simple random sampling is arguably the easiest to understand and implement sampling strategy, and it is statistically valid and suitable as long as there are no unbalance between classes or groups in the data; otherwise, the resulting sampling might not represent subgroups properly when the sample size is small.
Stratified sampling is another statistically robust approach that ensures total representation of subgroups that exist in the data, including minority ones. This is a common go-to approach when sampling from a large, labeled, fixed-size dataset of instances belonging to multiple classes to train, validate, and test a classification model.
Cluster sampling has a moderate level of randomness (not as high as in random and stratified sampling) and it is adequate for use cases involving large and geographically distributed populations, being more cost-efficient than other methods. Among its limitations, it tends to incur comparatively higher sampling error, and internal cluster diversity may not be guaranteed.
Systematic sampling is a common method to sample ordered data like time series. Despite its simplicity, it has the potential risk of the appearance of periodic patterns that may add bias to the resulting sample.
Reservoir sampling is utilized when the data has an unknown size or is being collected in a streaming fashion, being an efficient strategy for online or real-time data processing workflows. Unlike other methods, its mechanisms are not very intuitive, and its applicability is exclusively limited to the aforesaid scenarios.
When statistical robustness is not a top priority (sampling can be non-probability driven) and samples do not need to be forcibly bias-free, there are other sampling methods to consider:
- Convenience sampling is a low-randomness approach sometimes suitable for early-stage data collection or exploratory data analysis purposes. While it is an efficient and inexpensive method, it naturally brings a higher risk of bias than other methods, as it is not generalizable to a variety of scenarios.
- Purposive sampling, also called judgmental clustering, is resorted to when domain expertise is the key sampling factor. It can be based on expert knowledge or niche domain studies, but it is inherently subject to subjectivity, being also a not highly generalizable method. This method is used, for instance, when researchers need to “manually” pick certain subjects for a survey or experiment based on their specific attributes.
- Bootstrap sampling is a popular resampling technique with replacement that is used, for example, in the process of validating machine learning models. It is a powerful sampling approach when the data available is not sufficiently large, but it can cause overfitting if not used properly. Furthermore, the quality of the original data to sample from influences the quality of the sampling result.
Wrapping Up
This guide outlined eight popular data sampling methods in data science and machine learning workflows and presented a decision tree-based guide to help you choose the right one depending on the purpose, sampling goals, and characteristics of the data to sample on.
