Machine learning models can face problems such as overfitting and high variance. Overfitting occurs when a model fits the training data so well that it captures noise and random fluctuations rather than the underlying patterns. As a result, the model performs well on the training data but struggles to generalize to new, unseen data. High variance refers to the model’s sensitivity to small changes in the training data. This leads to a large variability in the predictions.
Bagging, short for bootstrap aggregating, is an ensemble learning technique designed to address these problems. It reduces overfitting and high variance by training multiple models on different subsets of the data and combining their predictions. In this article, we’ll explain how to perform bagging in R.
How Does Bagging Work?
Bagging uses bootstrapping, a method where random samples are taken from the dataset with replacement. Here’s how bagging works:
- Bootstrapping the Data: Create multiple bootstrap sample of the original dataset. Each sample has the same size as the original dataset but may contain duplicate records and omit others.
- Training Models: Train a separate base model on each bootstrap sample. These models are independent of one another. These models are independent of one another and learn unique patterns due to variations in the training subsets.
- Aggregating Predictions: Combine the predictions from all base models to produce a final prediction:
- For regression tasks: Aggregate the predictions by taking the average of the outputs from all base models
- For classification tasks: Aggregate the predictions using majority voting, where the most frequently predicted class is the final output
Install Required Libraries
R provides several libraries to implement bagging. Some of the most common ones are:
- ipred: Implements bagging for classification and regression tasks
- caret: Provides a unified interface for bagging and other machine learning techniques
Install these libraries if not already installed:
install.packages("ipred")
install.packages("caret")
After installing the libraries, you must load them into your R session.
# Load the ipred and caret packages library(ipred) library(caret)
Load the Data
Choose a dataset for your analysis. For this example, we’ll use the iris dataset, which is built into R.
data(iris) head(iris)
Split the Data
Split the data into training and testing sets.
set.seed(123) # For reproducibility trainIndex <- createDataPartition(iris$Species, p = 0.7, list = FALSE) trainData <- iris[trainIndex, ] testData <- iris[-trainIndex, ]
Perform Bagging Using the ipred Package
The ipred package is specifically designed for bagging. It provides functions to apply bagging to classification and regression tasks. The bagging() function in ipred is easy to use for creating robust models.
Parameters in ipred::bagging:
- nbagg: Number of bootstrap samples/models to aggregate
- coob: If TRUE, computes out-of-bag error for evaluation
# Train a bagging model
bag_model <- bagging(Species ~ ., data = trainData,
nbagg = 25, coob = TRUE)
# Print the model
bag_model
Output:
Bagging classification trees with 25 bootstrap replications
Call: bagging.data.frame(formula = Species ~ ., data = trainData,
nbagg = 25, coob = TRUE)
Out-of-bag estimate of misclassification error: 0.0476
The output shows a bagging model with 25 bootstrap samples. It classifies the Species variable. The out-of-bag (OOB) error is 0.0476. This means 4.76% of the samples were misclassified. The OOB error helps assess the model’s performance.
Perform Bagging Using the caret Package
The caret package makes it easy to use bagging and offers extra options to adjust the model for better performance. The treebag method in caret performs bagging for classification tasks.
Parameters in caret::train:
- method: The model type to be used
- trControl: Controls the resampling method, such as cross-validation or bootstrapping
# Train a bagging model bag_control <- trainControl(method = "boot", number = 25) caret_bag_model <- train(Species ~ ., data = trainData, method = "treebag", trControl = bag_control) # Make predictions caret_predictions <- predict(caret_bag_model, testData) # Evaluate accuracy confusionMatrix(caret_predictions, testData$Species)
Output:
Confusion Matrix and Statistics
Reference
Prediction setosa versicolor virginica
setosa 15 0 0
versicolor 0 14 2
virginica 0 1 13
Overall Statistics
Accuracy : 0.9333
95% CI : (0.8173, 0.986)
No Information Rate : 0.3333
P-Value [Acc > NIR] : < 2.2e-16
Kappa : 0.9
Mcnemar's Test P-Value : NA
Statistics by Class:
Class: setosa Class: versicolor Class: virginica
Sensitivity 1.0000 0.9333 0.8667
Specificity 1.0000 0.9333 0.9667
Pos Pred Value 1.0000 0.8750 0.9286
Neg Pred Value 1.0000 0.9655 0.9355
Prevalence 0.3333 0.3333 0.3333
Detection Rate 0.3333 0.3111 0.2889
Detection Prevalence 0.3333 0.3556 0.3111
Balanced Accuracy 1.0000 0.9333 0.9167
The output shows the confusion matrix for three classes: setosa, versicolor, and virginica. The model has an accuracy of 93.33%. Sensitivity and specificity are high across all classes. The positive and negative predictive values are good. Balanced accuracy is close to 1.
Conclusion
Bagging is a technique that helps improve the performance of models by making them more stable and reducing errors. It works by combining predictions from multiple models, which helps to make the final result more reliable. In R, you can use libraries like ipred and caret to apply bagging for classification and regression tasks. By adjusting hyperparaeters, you can improve your model and get the best performance.
