Image by Author | Ideogram
Missing data constitutes a common issue frequently found in many real-world data analysis scenarios, like financial trend analysis, weather monitoring for forecasting, or sensor data collection in smart homes, cities, and environments. Since many of these are high-stakes scenarios, leaving missing data “as it is” could be problematic and prone to misleading analysis results, particularly in series data where the sequential or temporal relationship between data entries is fundamental. Therefore, it is recommended to adopt strategies to replace those missing data points with actual entries that contribute to continuity in data and drive more accurate analysis and visualizations.
This article illustrates one strategy for imputing missing data points in series data: interpolation, aided by Python’s NumPy library. Off we go!
Step-by-Step Process
As usual, we start by importing the necessary libraries to perform interpolation on incomplete series data in Python.
import numpy as np import matplotlib.pyplot as plt
Next, the most important asset: is some data. For this tutorial, we will generate a synthetic set of sequential data that represents daily temperature readings over 10 days, alongside an array of associated timestamps numbered from 1 to 10 (well, from 0 to 9 to be precise). The data is stored in a numpy array for simplicity. But there’s a catch: three out of the ten daily temperature readings went missing!
time = np.arange(10) temperatures = np.array([20, 21, np.nan, 23, np.nan, 25, 26, np.nan, 28, 29])
Among the approaches NumPy provides for interpolating missing data entries, linear interpolation is one of the simplest yet most commonly used ones. The interpolation process takes two simple steps:
- Identify the missing indices or positions in the series where the data are missing.
- Perform the linear interpolation with NumPy’s
np.interp()function. The second and third arguments passed to this function contain the time instants (resp. associated temperature readings) that do exist in the series: the negation (~) operator inverts the indices identified in the previous step to extract only data in positions with known temperature readings in both arrays.
missing_indices = np.isnan(temperatures) interpolated_temperatures = np.interp(time, time[~missing_indices], temperatures[~missing_indices])
At this point, the interpolation is done and our series of temperature readings is complete. But because this is a “see-to-believe” story, let’s visualize the result.
plt.figure(figsize=(8,5))
plt.plot(time, temperatures, 'o', label='Original Series (With Missing Entries)')
plt.plot(time, interpolated_temperatures, '-', label='Interpolated Data')
plt.xlabel("Time")
plt.ylabel("Temperature (°C)")
plt.legend()
plt.title("Example of Linear Interpolation with NumPy")
plt.show()
That was a pretty easy example, but for a slightly more challenging data series, the same method should work.
On another note, there’s a similar and even more straightforward approach for data interpolation provided by other libraries like Pandas. Here’s an example:
import pandas as pd time = pd.Series(range(10)) temperatures = pd.Series([15.2, np.nan, 17.8, 20.1, np.nan, 22.5, 24.3, np.nan, 27.9, 30.4]) interpolated_temperatures = temperatures.interpolate(method='linear') print(interpolated_temperatures)
And its related visualization:
Wrapping Up
In sum, NumPy provides a simple yet effective linear interpolation method to complete series data that contain missing values. Importantly, the effectiveness of this method is normally limited to not very complex data, and other alternative methods provided by libraries like Pandas and SciPy (for nonlinear interpolation) shall be considered on more challenging datasets.
