To “detrend” time series data means to remove an underlying trend in the data. The main reason we would want to do this is to more easily see subtrends in the data that are seasonal or cyclical.
For example, consider the following time series data that represents the total sales for some company during 20 consecutive periods:
Clearly the sales is trending upwards over time, but there also appears to be a cyclical or seasonal trend in the data, which can be seen by the tiny “hills” that occur over time.
To gain a better view of this cyclical trend, we can detrend the data. In this case, this would involve removing the overall upward trend over time so that the resulting data represents just the cyclical trend.
There are two common methods used to detrend time series data:
1. Detrend by Differencing
2. Detrend by Model Fitting
This tutorial provides a brief explanation of each method.
Method 1: Detrend by Differencing
One way to detrend time series data is to simply create a new dataset where each observation is the difference between itself and the previous observation.
For example, the following image shows how to use differencing to detrend a data series.
To obtain the first value of the detrended time series data, we calculate 13 – 8 = 5. Then to obtain the next value we calculate 18-13 = 5, and so on.
The following plot shows the original time series data:
And this plot shows the detrended data:
Notice how it’s much easier to see the seasonal trend in the time series data in this plot because the overall upward trend has been removed.
Method 2: Detrend by Model Fitting
Another way to detrend time series data is to fit a regression model to the data and then calculate the difference between the observed values and the predicted values from the model.
For example, suppose we have the same dataset:
If we fit a simple linear regression model to the data, we can obtain a predicted value for each observation in the dataset.
We can then find the difference between the actual value and the predicted value for each observation. These differences represent the detrended data.
If we create a plot of the detrended data, we can visualize the seasonal or cyclical trend in the data much more easily:
Note that we used linear regression in this example, but it’s possible to use a more complex method like exponential regression if there is more of an exponentially increasing or decreasing trend in the data.
If all the articles here are similar to this well-written and reader-friendly article, I can learn statistics soon 🙂 Thanks
great explanation
Thank you, it is the most valuable information for freshers.
thank you very much for this post. it explicitly explained the concepts. no ambiguities. thanks alot
thank you so much! if you don’t mind, can I please ask for some book or article that prove why it is valid to use regression to detrend data?
Hi Ngoc…You are very welcome! Detrending data using regression is a common technique in time series analysis, and several resources discuss its validity and applications. Here are a few recommendations for books and articles that explain why regression is a valid method for detrending data:
### Books
1. **”Time Series Analysis: Forecasting and Control” by George E. P. Box, Gwilym M. Jenkins, Gregory C. Reinsel, and Greta M. Ljung**
– This book is a classic in time series analysis and covers various methods for detrending data, including the use of regression.
2. **”Applied Time Series Analysis” by Terence C. Mills**
– This book provides practical applications of time series analysis techniques, including regression methods for detrending data.
3. **”Time Series Analysis by State Space Methods” by James Durbin and Siem Jan Koopman**
– This book includes discussions on state-space models and regression methods for handling trends in time series data.
### Articles
1. **”A Simple Introduction to Time Series Analysis” by Robert Nau**
– This online article provides a straightforward introduction to time series analysis and includes sections on detrending using regression. [Read it here](https://people.duke.edu/~rnau/411arim.htm).
2. **”Detrending Time Series: A Cautionary Note” by Alan T. Peacock and William F. Scarth**
– This paper discusses various methods for detrending, including regression, and provides insights into their validity and potential pitfalls.
3. **”Time Series Analysis: With Applications in R” by Jonathan D. Cryer and Kung-Sik Chan**
– This book has a comprehensive chapter on detrending and includes practical examples using regression techniques. [Preview on Google Books](https://books.google.com).
4. **”An Introduction to Time Series Regression and Forecasting” by Robert Nau**
– This is a series of lecture notes that explain the principles of time series regression and its applications in forecasting, including detrending. [Read it here](https://people.duke.edu/~rnau/regintro.htm).
### Key Points on Why Regression is Valid for Detrending
1. **Linear Trends**: Regression is particularly useful for identifying and removing linear trends in time series data. By fitting a regression line, the deterministic component can be separated from the stochastic component.
2. **Statistical Properties**: Regression-based detrending helps maintain the statistical properties of the residual series, which is essential for accurate modeling and forecasting.
3. **Flexibility**: Regression methods can be extended to handle non-linear trends using polynomial regression or other non-linear models, making it a versatile tool for detrending.
4. **Simplicity and Interpretability**: Regression models are straightforward to implement and interpret, providing clear insights into the trend component of the time series.
These resources should provide a solid foundation for understanding the theoretical and practical aspects of using regression to detrend data.