PySpark: How to Add Months to a Date Column


You can use the following syntax to add a specific number of months to a date column in a PySpark DataFrame:

from pyspark.sql import functions as F

df.withColumn('add5months', F.add_months(df['date'], 5)).show()

This particular example creates a new column called add5months that adds 5 months to each date in the date column.

The following example shows how to use this syntax in practice.

Example: How to Add Months to a Date Column in PySpark

Suppose we have the following PySpark DataFrame that contains information about sales made on various dates at some company:

from pyspark.sql import SparkSession
spark = SparkSession.builder.getOrCreate()

#define data
data = [['2023-01-15', 225],
        ['2023-02-24', 260],
        ['2023-07-14', 413],
        ['2023-10-30', 368],
        ['2023-11-03', 322],
        ['2023-11-26', 278]] 
  
#define column names
columns = ['date', 'sales'] 
  
#create dataframe using data and column names
df = spark.createDataFrame(data, columns) 
  
#view dataframe
df.show()

+----------+-----+
|      date|sales|
+----------+-----+
|2023-01-15|  225|
|2023-02-24|  260|
|2023-07-14|  413|
|2023-10-30|  368|
|2023-11-03|  322|
|2023-11-26|  278|
+----------+-----+

Suppose we would like to add a new column that adds 5 months to each date in the date column.

We can use the following syntax to do so:

from pyspark.sql import functions as F

#add 5 months to each date in 'date' column
df.withColumn('add5months', F.add_months(df['date'], 5)).show()

+----------+-----+----------+
|      date|sales|add5months|
+----------+-----+----------+
|2023-01-15|  225|2023-06-15|
|2023-02-24|  260|2023-07-24|
|2023-07-14|  413|2023-12-14|
|2023-10-30|  368|2024-03-30|
|2023-11-03|  322|2024-04-03|
|2023-11-26|  278|2024-04-26|
+----------+-----+----------+

Notice that the new add5months column contains each of the dates from the date column with five months added.

Note that if you would instead like to subtract 5 months, you could use a value of -5 in the add_months() function instead:

from pyspark.sql import functions as F

#subtract 5 months from each date in 'date' column
df.withColumn('sub5months', F.add_months(df['date'], -5)).show()

+----------+-----+----------+
|      date|sales|sub5months|
+----------+-----+----------+
|2023-01-15|  225|2022-08-15|
|2023-02-24|  260|2022-09-24|
|2023-07-14|  413|2023-02-14|
|2023-10-30|  368|2023-05-30|
|2023-11-03|  322|2023-06-03|
|2023-11-26|  278|2023-06-26|
+----------+-----+----------+

Notice that the new sub5months column contains each of the dates from the date column with five months subtracted.

Note that we used the withColumn function to return a new DataFrame with the sub5months column added and all other columns left the same.

You can find the complete documentation for the PySpark withColumn function here.

Additional Resources

The following tutorials explain how to perform other common tasks in PySpark:

How to Add Days to a Date Column in PySpark
How to Convert String to Date in PySpark
How to Convert Timestamp to Date in PySpark

Featured Posts

Leave a Reply

Your email address will not be published. Required fields are marked *