How to Use the notna() Function in Pandas


Often you may want to check if a particular element in a pandas DataFrame is not missing.

The most efficient way to do so is by using the notna() function, which uses the following syntax:

DataFrame.notna()

This function will check if each element in a pandas Series or column of a pandas DataFrame is missing and return either True or False as the result.

The following example shows how to use the notna() function in practice with a pandas DataFrame.

Example: How to Use the notna() Function in Pandas

Suppose we create the following pandas DataFrame that contains information about various basketball players:

import pandas as pd
import numpy as np

#create DataFrame
df = pd.DataFrame({'team': ['A', 'A', 'B', 'B', 'C', 'None', 'C'],
                   'points': [12, 18, 18, 22, 30, 41, np.nan],
                   'assists': [np.nan, 10, np.nan, 11, 7, 12, 8]})

#view DataFrame
print(df)

   team  points  assists
0     A    12.0      NaN
1     A    18.0     10.0
2     B    18.0      NaN
3     B    22.0     11.0
4     C    30.0      7.0
5  None    41.0     12.0
6     C     NaN      8.0

Suppose that we would like to check if each value in each column of the DataFrame is not missing.

We can use the following syntax with the DataFrame.notna() function to do so:

#check if each value in each column is not missing
df.notna()

	team	points	assists
0	True	True	False
1	True	True	True
2	True	True	False
3	True	True	True
4	True	True	True
5	True	True	True
6	True	False	True

The output displays either True or False for each value in each row of the DataFrame.

Here is how to interpret the output:

  • A value of True indicates that the element is not missing.
  • A value of False indicates that the element is missing.

In this particular DataFrame, we intentionally inserted several NaN values by using np.nan from the NumPy package.

Notice that each of the False values that are returned from the notna() function correspond to elements in the pandas DataFrame that contain np.nan values.

Note: The one string value of ‘None’ in the team column of the DataFrame is not considered to be a missing value because it represents a string.

Note that we can also check if the values in one particular column of the DataFrame are missing or not.

For example, we can use the following syntax to check if each value in the assists column is missing:

#check if each value in 'assists' column is not missing
df['assists'].notna()

0    False
1     True
2    False
3     True
4     True
5     True
6     True
Name: assists, dtype: bool

Notice that this returns True and False values for each value in the assists column only.

Also note that you can use the notna() function along with the sum() function to find the total number of non-missing values in a column.

For example, we can use the following syntax to count how many elements are not missing in the assists column:

#count non-missing values in 'assists' column
df['assists'].notna().sum()

5

The output returns a value of 5, which tells us that there are 5 non-missing values in the assists column.

Note: You can find the complete documentation for the notna() function in pandas here.

Additional Resources

The following tutorials explain how to perform other common tasks in pandas:

How to Use the nunique() Function in Pandas
How to Use the get_loc() Function in Pandas
How to Create a Tuple from Two Columns in Pandas

Featured Posts

Leave a Reply

Your email address will not be published. Required fields are marked *