How to Normalize Columns in a Pandas DataFrame


Often you may want to normalize the data values of one or more columns in a pandas DataFrame.

This tutorial explains two ways to do so:

1. Min-Max Normalization

  • Objective: Converts each data value to a value between 0 and 1.
  • Formula: New value = (value – min) / (max – min)

2. Mean Normalization

  • Objective: Scales values such that the mean of all values is 0 and std. dev. is 1. 
  • Formula: New value = (value – mean) / (standard deviation)

Let’s check out an example of how to use each method on a pandas DataFrame.

Example 1: Min-Max Normalization

Suppose we have the following pandas DataFrame:

import pandas as pd

#create DataFrame
df = pd.DataFrame({'points': [25, 12, 15, 14, 19],
                   'assists': [5, 7, 7, 9, 12],
                   'rebounds': [11, 8, 10, 6, 6]}) 

#view DataFrame
print(df)

   points  assists  rebounds
0      25        5        11
1      12        7         8
2      15        7        10
3      14        9         6
4      19       12         6

We can use the following code to apply a min-max normalization to each column in the DataFrame:

(df-df.min())/(df.max()-df.min())

        points	        assists	        rebounds
0	1.000000	0.000000	1.0
1	0.000000	0.285714	0.4
2	0.230769	0.285714	0.8
3	0.153846	0.571429	0.0
4	0.538462	1.000000	0.0

The max value in each column is now equal to and the min value in each column is now equal to 0, with all other values ranging between 0 and 1.

Example 2: Mean Normalization

Once again suppose we have the following pandas DataFrame:

import pandas as pd

#create DataFrame
df = pd.DataFrame({'points': [25, 12, 15, 14, 19],
                   'assists': [5, 7, 7, 9, 12],
                   'rebounds': [11, 8, 10, 6, 6]}) 

#view DataFrame
print(df)

   points  assists  rebounds
0      25        5        11
1      12        7         8
2      15        7        10
3      14        9         6
4      19       12         6

We can use the following code to apply a mean normalization to each column in the DataFrame:

(df-df.mean())/df.std()

        points	        assists   	rebounds
0	1.554057	-1.133893	1.227881
1	-0.971286	-0.377964	-0.087706
2	-0.388514	-0.377964	0.789352
3	-0.582772	0.377964	-0.964764
4	0.388514	1.511858	-0.964764

The values in each column are now normalized such that the mean of the values in each column is 0 and the standard deviation of values in each column is 1.

If a particular data point has a normalized value greater than 0, it’s an indication that the data point is greater than the mean of its column. Conversely, a normalized value less than 0 is an indication that the data point is less than the mean of its column.

Additional Resources

Pandas: How to Group and Aggregate by Multiple Columns
How to Filter a Pandas DataFrame on Multiple Conditions
How to Count Missing Values in a Pandas DataFrame

Leave a Reply

Your email address will not be published.