Missing Value Imputation Methods using Python

Missing values occur in real-world datasets for many reasons: data-entry errors, survey non-response, equipment failures, and corruption are only a few examples. Missing-value imputation replaces those absent observations with substituted values so that an analysis or model can continue.

This guide covers three common groups of imputation techniques and implements each one with Python:

1. Mean, median, and mode imputation
2. Predictive imputation
3. Last Observation Carried Forward (LOCF) and Next Observation Carried Backward (NOCB)

Mean, median, and mode imputation

These statistical methods replace missing values with a summary of the observed values in a feature.

• Mean imputation uses the arithmetic average. It is most appropriate for numerical data without influential outliers because extreme values can shift the mean substantially.
• Median imputation uses the middle observed value. It is more robust for skewed distributions or data with outliers.
• Mode imputation uses the most frequent value and is commonly applied to categorical data.

The following example applies all three methods to the same feature:

import numpy as np
import pandas as pd
 
data = {'Score': [25, np.nan, 30, np.nan, 29, 27, 32, 31]}
df = pd.DataFrame(data)
 
df['Score_Mean'] = df['Score'].fillna(df['Score'].mean())
df['Score_Median'] = df['Score'].fillna(df['Score'].median())
df['Score_Mode'] = df['Score'].fillna(df['Score'].mode()[0])
 
print(df)

   Score  Score_Mean  Score_Median  Score_Mode
0   25.0        25.0          25.0        25.0
1    NaN        29.0          29.5        25.0
2   30.0        30.0          30.0        30.0
3    NaN        29.0          29.5        25.0
4   29.0        29.0          29.0        29.0
5   27.0        27.0          27.0        27.0
6   32.0        32.0          32.0        32.0
7   31.0        31.0          31.0        31.0

These methods are fast and simple, but they can bias estimates when values are not missing at random. They also reduce a feature's variance, which can lead to underestimated standard errors.

Predictive imputation

Predictive imputation estimates missing values from relationships in the rest of the data. Common approaches include:

• Regression imputation: trains a regression model on related variables and uses its predictions to fill the missing values.
• K-nearest neighbors imputation: finds the $k$ observations most similar to the incomplete row and imputes from their values.

Scikit-learn's KNNImputer provides a direct implementation:

import numpy as np
import pandas as pd
from sklearn.impute import KNNImputer
 
data = {
    'Feature1': [25, 20, 30, 40, 29, 27, 32, 31],
    'Feature2': [20, 25, np.nan, 45, 30, 25, 35, 40],
}
df = pd.DataFrame(data)
 
imputer = KNNImputer(n_neighbors=2)
df_filled = imputer.fit_transform(df)
 
print(df_filled)

[[25. 20.]
 [20. 25.]
 [30. 35.]
 [40. 45.]
 [29. 30.]
 [27. 25.]
 [32. 35.]
 [31. 40.]]

Predictive methods can preserve complex relationships better than simple summary statistics, but their quality still depends on the available features, preprocessing, and assumptions of the selected model.

LOCF and NOCB

Last Observation Carried Forward and Next Observation Carried Backward are intended for ordered observations such as time-series or longitudinal data.

• LOCF fills a missing value with the most recently observed value before it.
• NOCB fills a missing value with the next observed value after it.

Pandas provides ffill() and bfill() for these operations:

import numpy as np
import pandas as pd
 
time_data = {
    'Time': pd.date_range(start='2023-01-01', periods=8, freq='D'),
    'Value': [1, np.nan, np.nan, 4, 5, np.nan, 7, 8],
}
df_time = pd.DataFrame(time_data)
 
df_time['Value_LOCF'] = df_time['Value'].ffill()
df_time['Value_NOCB'] = df_time['Value'].bfill()
 
print(df_time)

        Time  Value  Value_LOCF  Value_NOCB
0 2023-01-01    1.0         1.0         1.0
1 2023-01-02    NaN         1.0         4.0
2 2023-01-03    NaN         1.0         4.0
3 2023-01-04    4.0         4.0         4.0
4 2023-01-05    5.0         5.0         5.0
5 2023-01-06    NaN         5.0         7.0
6 2023-01-07    7.0         7.0         7.0
7 2023-01-08    8.0         8.0         8.0

LOCF and NOCB are straightforward, but they can introduce significant bias and underestimate variability. The risk is especially high when a time series has a trend or when observations are not missing at random.

How to choose an imputation technique

Choose a method according to the feature's distribution, the missingness mechanism, the relationships among features, and the role that ordering plays in the data.

Mean imputation may be unsuitable for skewed data or data with outliers. LOCF and NOCB can distort a time series with a strong trend or seasonality. Predictive methods may perform better, but they introduce model assumptions and must be evaluated without leaking information from validation or test data.

Always begin with exploratory analysis. Measure how much data is missing, look for patterns in the missingness, inspect the observed distributions, and compare plausible techniques against the downstream objective.

Summary

• Mean, median, and mode imputation provide simple baselines for numerical and categorical features.
• Predictive imputation uses relationships among features to estimate absent values.
• LOCF and NOCB propagate nearby observations through ordered data.

No method is universally correct. The best choice is the one whose assumptions fit the data and whose effect is validated in the full analytical or modeling workflow.