Duplicate Data
Duplicate data means the same record appears more than once in a dataset when those records represent the same real-world example.
One real-world record should not accidentally appear multiple times.
Duplicate records can make some examples appear more important than they really are. Before training a model, we should check whether duplicate records exist and decide how to handle them.
What Does Duplicate Data Look Like?
Imagine we have customer information:
The first and second rows contain exactly the same information.
If both rows represent the same customer record, the second row is a duplicate.
Why Can Duplicate Data Be a Problem?
Machine learning models learn from the examples in the training dataset.
If the same example appears many times accidentally, that example can have more influence than it should.
Appears once.
Appears multiple times.
The model may effectively see Customer A as several training examples even though there is only one real customer record.
A Simple Example
Suppose we are building a model to predict whether a customer will purchase a product.
Orders: 5
Visits: 12
Orders: 5
Visits: 12
If these two rows represent the same customer and were accidentally imported twice, the dataset now contains incorrect duplicate information.
The problem is not that Rahul is a customer. The problem is that the same record was counted twice.
How Do Duplicates Happen?
Duplicate records can appear for several practical reasons.
Duplicate Does Not Always Mean "Same Name"
This is an important point.
Two records with the same name are not automatically duplicates.
Hyderabad
Age: 29
Chennai
Age: 41
These records have the same name, but they may represent two different people.
What Should We Do With Duplicates?
If we confirm that two records represent the same real-world example and one is an accidental duplicate, we can remove the unnecessary duplicate.
Record 1
Record 2
Keep the appropriate record.
But don't blindly delete records just because some values look similar. First determine whether they really represent the same observation.
Real-World Example
Imagine an e-commerce company combines customer data from two systems.
Customer ID 1001 appears twice with the same customer information.
This could be a duplicate created when the two systems were combined.
Before training the model, the data team should investigate the duplicate and decide which record should remain.
One Real-World Example Should Not Be Accidentally Counted Multiple Times.
Duplicate data can give some examples too much influence during training. Identify genuine duplicates and handle them before building the model.
Are These Records Duplicates?
The first two records are likely duplicates if they represent the same person and the same observation. The third record is not automatically a duplicate just because the name is also Rahul.