Bootstrapping is a method that can be used to estimate the standard error of any statistic and produce a confidence interval for the statistic.
The basic process for bootstrapping is as follows:
- Take k repeated samples with replacement from a given dataset.
- For each sample, calculate the statistic you’re interested in.
- This results in k different estimates for a given statistic, which you can then use to calculate the standard error of the statistic and create a confidence interval for the statistic.
We can perform bootstrapping in R by using the following functions from the boot library:
1. Generate bootstrap samples.
boot(data, statistic, R, …)
where:
- data: A vector, matrix, or data frame
- statistic: A function that produces the statistic(s) to be bootstrapped
- R: Number of bootstrap replicates
2. Generate a bootstrapped confidence interval.
boot.ci(bootobject, conf, type)
where:
- bootobject: An object returned by the boot() function
- conf: The confidence interval to calculate. Default is 0.95
- type: Type of confidence interval to calculate. Options include “norm”, “basic”, “stud”, “perc”, “bca” and “all” – Default is “all”
The following examples show how to use these functions in practice.
Example 1: Bootstrap a Single Statistic
The following code shows how to calculate the standard error for the R-squared of a simple linear regression model:
set.seed(0) library(boot) #define function to calculate R-squared rsq_function <- function(formula, data, indices) { d <- data[indices,] #allows boot to select sample fit <- lm(formula, data=d) #fit regression model return(summary(fit)$r.square) #return R-squared of model } #perform bootstrapping with 2000 replications reps <- boot(data=mtcars, statistic=rsq_function, R=2000, formula=mpg~disp) #view results of boostrapping reps ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = mtcars, statistic = rsq_function, R = 2000, formula = mpg ~ disp) Bootstrap Statistics : original bias std. error t1* 0.7183433 0.002164339 0.06513426
From the results we can see:
- The estimated R-squared for this regression model is 0.7183433.
- The standard error for this estimate is 0.06513426.
We can quickly view the distribution of the bootstrapped samples as well:
plot(reps)
We can also use the following code to calculate the 95% confidence interval for the estimated R-squared of the model:
#calculate adjusted bootstrap percentile (BCa) interval boot.ci(reps, type="bca") CALL : boot.ci(boot.out = reps, type = "bca") Intervals : Level BCa 95% ( 0.5350, 0.8188 ) Calculations and Intervals on Original Scale
From the output we can see that the 95% bootstrapped confidence interval for the true R-squared values is (.5350, .8188).
Example 2: Bootstrap Multiple Statistics
The following code shows how to calculate the standard error for each coefficient in a multiple linear regression model:
set.seed(0) library(boot) #define function to calculate fitted regression coefficients coef_function <- function(formula, data, indices) { d <- data[indices,] #allows boot to select sample fit <- lm(formula, data=d) #fit regression model return(coef(fit)) #return coefficient estimates of model } #perform bootstrapping with 2000 replications reps <- boot(data=mtcars, statistic=coef_function, R=2000, formula=mpg~disp) #view results of boostrapping reps ORDINARY NONPARAMETRIC BOOTSTRAP Call: boot(data = mtcars, statistic = coef_function, R = 2000, formula = mpg ~ disp) Bootstrap Statistics : original bias std. error t1* 29.59985476 -5.058601e-02 1.49354577 t2* -0.04121512 6.549384e-05 0.00527082
From the results we can see:
- The estimated coefficient for the intercept of the model is 29.59985476 and the standard error of this estimate is 1.49354577.
- The estimated coefficient for the predictor variable disp in the model is -0.04121512 and the standard error of this estimate is 0.00527082.
We can quickly view the distribution of the bootstrapped samples as well:
plot(reps, index=1) #intercept of model plot(reps, index=2) #disp predictor variable
We can also use the following code to calculate the 95% confidence intervals for each coefficient:
#calculate adjusted bootstrap percentile (BCa) intervals boot.ci(reps, type="bca", index=1) #intercept of model boot.ci(reps, type="bca", index=2) #disp predictor variable CALL : boot.ci(boot.out = reps, type = "bca", index = 1) Intervals : Level BCa 95% (26.78, 32.66 ) Calculations and Intervals on Original Scale BOOTSTRAP CONFIDENCE INTERVAL CALCULATIONS Based on 2000 bootstrap replicates CALL : boot.ci(boot.out = reps, type = "bca", index = 2) Intervals : Level BCa 95% (-0.0520, -0.0312 ) Calculations and Intervals on Original Scale
From the output we can see that the 95% bootstrapped confidence intervals for the model coefficients are as follows:
- C.I. for intercept: (26.78, 32.66)
- C.I. for disp: (-.0520, -.0312)
Additional Resources
How to Perform Simple Linear Regression in R
How to Perform Multiple Linear Regression in R
Introduction to Confidence Intervals
Hello, thank you for the code ! How can I get the information for F-statistic, df and p-value of R2? I need these information for reporting the results of the regression in my thesis. Because normally when reporting regression results its like this:
R2 = .10, F(3, 210) = 6.22, p < .01
β=−.210, p<.05
Thank you for your reply!
Kind regards, Anna-Lena
Can I resampling a frequency distribution (length: frequency) and calculate maximum length
Hi Imam…Yes, you can perform bootstrapping on a frequency distribution in R and calculate the maximum length for each resample. Here’s a step-by-step guide to achieve this:
1. **Create the Frequency Distribution:**
Suppose you have a frequency distribution of lengths. For simplicity, let’s assume it’s stored in a data frame `df` with columns `length` and `frequency`.
2. **Resampling:**
To perform bootstrapping, you’ll need to resample from your frequency distribution. You’ll sample with replacement to create a new dataset of the same size.
3. **Calculate Maximum Length:**
For each bootstrap sample, calculate the maximum length.
4. **Repeat the Process:**
Repeat the resampling and calculation steps many times (e.g., 1000 times) to get a distribution of the maximum lengths.
Here is an example in R:
“`r
# Sample data
df <- data.frame( length = c(5, 10, 15, 20, 25), frequency = c(10, 5, 3, 2, 1) ) # Convert the frequency distribution to a vector of lengths lengths <- rep(df$length, df$frequency) # Number of bootstrap samples n_bootstrap <- 1000 # Function to perform one bootstrap resample and calculate the max length bootstrap_max <- function(data) { sample_data <- sample(data, length(data), replace = TRUE) return(max(sample_data)) } # Perform bootstrapping bootstrap_results <- replicate(n_bootstrap, bootstrap_max(lengths)) # Convert results to a data frame for easier manipulation bootstrap_results_df <- data.frame(max_length = bootstrap_results) # Summary of bootstrap results summary(bootstrap_results_df) ``` ### Explanation: 1. **Data Preparation:** - `df` is your frequency distribution data frame. - `lengths` converts the frequency distribution into a vector where each length appears according to its frequency. 2. **Bootstrap Resampling:** - `n_bootstrap` specifies the number of bootstrap samples. - `bootstrap_max` function performs a single bootstrap resample and calculates the maximum length. 3. **Perform Bootstrapping:** - `replicate` performs the bootstrap resampling and maximum length calculation `n_bootstrap` times. - The results are stored in `bootstrap_results`. 4. **Summary:** - `summary(bootstrap_results_df)` provides a summary of the distribution of the maximum lengths from the bootstrap samples. This method allows you to understand the variability in the maximum length estimation based on your frequency distribution.
Would prefer you started with simpler example whereby you present a simple vector and calculate, say, 95% ci of mean. Using the function command adds an unnecessary layer of complexity that makes it diificult to understand.
But thanks anyway
jeff
Absolutely — let’s start **super simple** with a plain numeric vector and calculate a **95% confidence interval (CI) of the mean** using **bootstrapping** in R — **without using custom functions**, just enough code to make the concept click.
—
### 🔢 Step 1: Create a simple vector
“`r
data <- c(5, 7, 9, 10, 12, 13, 15) ``` This is your sample data — say, test scores of 7 students. --- ### 🎲 Step 2: Perform bootstrapping manually We’ll resample the data **with replacement**, say **1000 times**, and calculate the mean each time. ```r set.seed(123) # for reproducibility boot_means <- replicate(1000, mean(sample(data, replace = TRUE))) ``` - `sample(data, replace = TRUE)` takes a random sample **with replacement**. - `mean(...)` computes the sample mean. - `replicate(1000, ...)` repeats this 1000 times to generate bootstrap means. --- ### 📏 Step 3: Calculate the 95% confidence interval ```r quantile(boot_means, c(0.025, 0.975)) ``` This gives you the **2.5th** and **97.5th** percentiles — the 95% CI for the mean. --- ### 📊 Optional: Visualize You can also plot a histogram: ```r hist(boot_means, main = "Bootstrap Distribution of the Mean", xlab = "Mean") abline(v = quantile(boot_means, c(0.025, 0.975)), col = "red", lty = 2) ``` --- ### 🧠 Summary - **No function** used — just basic `sample`, `mean`, `replicate`, and `quantile`. - You get a solid idea of how bootstrapping builds a distribution of a statistic (like the mean) and how confidence intervals are derived from it.