n8n remove duplicates node
Duplicate data can break automations, inflate analytics, or cause repeated actions like sending mail the same mail multiple times.
The n8n Remove node is a strong tool that can help you automatically identify and remove repeated items, both within a single execution and across multiple executions over time.
Before we start, if you are looking for help with a n8n project, we are taking on customers. Head over to our n8n Automation Engineer page.
What the Remove Duplicates Node Does
The Remove Duplicates node helps you identify and delete items that are:
identical across all fields or a subset of fields in a single execution
identical to, or surpassed by, items seen in previous executions
This is especially useful for:
Preventing users from creating multiple accounts
Avoiding duplicate customer orders or form submissions
Ensuring unique records in large datasets
Filtering out already-processed items from API feeds or databases
By comparing current items to previous executions, the node can:
Remove data that has already been seen, or
Keep only the newest or highest-value data.
Operation Modes
The node’s behavior depends on the Operation parameter. It offers three modes:
Remove Items Repeated Within Current Input
Identify and remove duplicate items within the current input, comparing across all fields or selected fields.Remove Items Processed in Previous Executions
Compare current items to items processed in earlier executions and remove duplicates.Clear Deduplication History
Clear the stored memory of previously seen items.
Remove Items Repeated Within Current Input
Use this when you only want to check for duplicates within the same execution.
For example: multiple identical rows in an uploaded CSV file, or repeated entries fetched from an API call.
Parameters
Compare: Controls which fields n8n compares when checking for duplicates.
All Fields – Compare every field in each item.
All Fields Except – Exclude specific fields (comma-separated).
Selected Fields – Include only specified fields (comma-separated).
Options
If you choose All Fields Except or Selected Fields, you’ll see these options:
Disable Dot Notation:
When enabled, n8n treats nested objects as raw text rather than dot-separated paths (e.g.user.address.city).Remove Other Fields:
Removes all fields that aren’t used in the comparison, leaving only the fields that were used to determine uniqueness.
Remove Items Processed in Previous Executions
This mode compares the current input to stored deduplication data from previous runs.
It’s perfect for workflows that run repeatedly — for example, fetching API results every hour or pulling RSS feed items daily.
Parameters
Keep Items Where: Defines how n8n decides which items to keep.
Value Is New: Remove items that have appeared before.
Value Is Higher than Any Previous Value: Keep only items with higher numeric values.
Value Is a Date Later than Any Previous Date: Keep only items with later (newer) dates.
Value to Dedupe On:
The field(s) used for comparison — this must correspond to the “Keep Items Where” option:For Value Is New: Use a unique ID (e.g.,
id,uuid, oremail).For Value Is Higher than Any Previous Value: Use a numeric or version field.
For Value Is a Date Later than Any Previous Date: Use a date field (ISO format).
Options
Scope:
Node (default) – Deduplication data is stored only for this specific node instance.
Workflow – Deduplication data is shared across all Remove Duplicates nodes in the same workflow.
History Size (available when using “Value Is New”):
Defines how many past items n8n remembers to compare against.
Default: 10,000 items
The scope determines whether this history is per node or shared across the workflow.
Clear Deduplication History
This mode doesn’t process items.
Instead, it manages or clears the stored deduplication database used by “Remove Items Processed in Previous Executions.”
Parameters
Mode:
Clean Database – Deletes all stored duplication data, resetting the dedupe history.
Options
Scope:
Node (default) – Clears deduplication data for this specific node.
Workflow – Clears deduplication data for all Remove Duplicates nodes that share workflow scope.
Example 1
Show Comparing all fields from a spreadsheet
We start by adding a Get rows in sheet node
Next we add the Remove duplicate node, and add the following parameters:
Operation: Remove Items Repeated Within Current Input.
Compare: All Fields
full flow
Example 2
Show Comparing specific field from spreadsheet
+ Remove other Fields
we start by addding the Get rows in sheet node. we get all the state city from a google sheet
Next, we add the Remove duplicate node with the following parameters:
full flow
Example 3
Here, we merge two post.
we start with a two code node that output data
second code node
Next, we add the Merge node and append the output of the two code node together
Next, we add the remove duplicate nodes and we remove repeated items in the current input from the merge node
full flow
Example 4
In this example, let’s remove items processed in previous executions
Next, we add the split out node that splits the array into individual emails
Next, we add the remove duplicate node. and remove items processed in previous executions.
full flow
Final thought
The Remove Duplicates node in n8n is used to detect and remove duplicate items in a workflow. It ensures that only unique data passes through to the next node , helpful when dealing with repeated records or API responses that contain duplicates.
Thank you for reading this article. Make sure to check out our other n8n content on the website. If you need any help with n8n workflows we are taking on customers so reach out
