PySpark: How to Extract Year from Date

You can use the following syntax to extract the year from a date in a PySpark DataFrame:

from pyspark.sql.functions import year

df_new = df.withColumn('year', year(df['date']))

This particular example creates a new column called year that extracts the year from the date in the date column.

The following example shows how to use this syntax in practice.

Example: How to Extract Year from Date in PySpark

Suppose we have the following PySpark DataFrame that contains information about the sales made on various days at some company:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

#define data
data = [['2021-04-11', 22],
        ['2021-04-15', 14],
        ['2021-04-17', 12],
        ['2022-05-21', 15],
        ['2022-05-23', 30],
        ['2023-10-26', 45],
        ['2023-10-28', 32],
        ['2023-10-29', 47]]
  
#define column names
columns = ['date', 'sales']
  
#create dataframe using data and column names
df = spark.createDataFrame(data, columns) 
  
#view dataframe
df.show()

+----------+-----+
|      date|sales|
+----------+-----+
|2021-04-11|   22|
|2021-04-15|   14|
|2021-04-17|   12|
|2022-05-21|   15|
|2022-05-23|   30|
|2023-10-26|   45|
|2023-10-28|   32|
|2023-10-29|   47|
+----------+-----+

Suppose we would like to extract the year from each date in the date column.

We can use the following syntax to do so:

from pyspark.sql.functions import year

#extract year from date column
df_new = df.withColumn('year', year(df['date']))

#view new DataFrame
df_new.show()

+----------+-----+----+
|      date|sales|year|
+----------+-----+----+
|2021-04-11|   22|2021|
|2021-04-15|   14|2021|
|2021-04-17|   12|2021|
|2022-05-21|   15|2022|
|2022-05-23|   30|2022|
|2023-10-26|   45|2023|
|2023-10-28|   32|2023|
|2023-10-29|   47|2023|
+----------+-----+----+

The new year column contains the year of each date in the date column.

Note that we used the withColumn function to add a new column called year to the DataFrame while keeping all existing columns the same.

Note: You can find the complete documentation for the PySpark withColumn function here.