Pandas Shift
Pandas shift allows you to compare previous or future rows or columns within a dataframe. This tutorial is based on a YouTube video published to our channel. If you want a video walk through check that out.
Python Pandas Shift YouTube Video
To start we’re going to create a simple dataframe in python:
import pandas as pd basic = pd.DataFrame({ "NumA": [121,23434,3544,6534,5635,636,7634,834], "NumB": [15401,124201, 452111,152441,452111, 101451, 254132, 452141], }) basic.head(10) 
Basic Dataframe Head
Shift 1 Row
To shift down 1 row, we pass the parameter of 1.
basic.head(10).shift(1) 
Shift 2 Row
To shift down 2 rows, we pass the parameter of 1.
basic.head(10).shift(2) 
Shift Backwards
To shift in the opposite direction, we invert the number passed.
basic.head(10).shift(-2) 
How to fill values
Often when shifting, you’ll encounter null values. These values will show up as NaN in your dataframe. This is the default when using shift.
To fill these in, use the parameter fill_value = YOUR_VALUE. Your Value is what you want inserted instead of a null value.Â
basic.head(10).shift(2, fill_value=0) 
Shift to the right or left
By default (and most of the time you’ll be using shift) shift looks at the row above or below. If you set the axis=1 as a parameter, you can look at columns.
basic.head(10).shift(1, axis = 1) 
Inversing the number will have the same impact as when we looked at reversing the rows above.
basic.head(10).shift(-1, axis = 1) 
Coding Interview Example
Let’s take a look at an example of how this can be used within a coding interview as it’s a quite common question.
Question: Find transactions where person paid more than last time. In the final dataframe get the previous and next transactions as well.
df = pd.DataFrame({ "Transaction_id": [1,2,3,4,5,6,7,8], "Person_id": [101,101, 111,141,111, 101, 132, 141], "Amount": [15, 25, 10, 85, 20, 45, 30, 15] }) df.head(10) Let’s use the dataframe from above to answer this question.
Let’s first sort the dataframe by the Person_id
df = df.sort_values('Person_id', ascending = True) df.head(10) 
Next we will groupby the person_id and shift the amount by 1 row
df['prev_amount'] = df.groupby('Person_id')['Amount'].shift(1) df.head(10) 
If you want to replicate this for the other direction, repeat the code. The only difference will be the -1 value.Â
df['next_amount'] = df.groupby('Person_id')['Amount'].shift(-1) df.head(10) 
We only care about the instance where the amount is greater than the previous amount, so filter the dataframe on that.
df.query('Amount > prev_amount') 
Lastly, we find the difference by creating a new column and taking amount – prev amount.
df['diff'] = df['Amount'] - df['prev_amount'] df.query('Amount > prev_amount') 
