Hello all, we’re dealing with a customer base that has a tendency to create multiple accounts. Sometimes when they’re trying to game the system and use a one-use-discount more than one time, but most often because they’re forgetful.
I know we can now merge customers when we know they have multiple profiles, but how do we go about finding duplicates in our customer base when we don’t know who we’re looking for?
I don’t want to sit here searching random names, so how can I filter “find all matching First name/Last name” or “find all with the same address” and get a list of all profiles that have matching first and last names, or matching names and addresses?
One approach for this problem is to use LLMs. LLMs allow you to do semantic matching rather than just string matching.
They can catch nuanced cases that rule-based filtering misses, like recognising “J. Smith” as “John Smith” or that “123 Main St Apt 4” is the same address as “123 Main Street #4”.
Here’s an example on 200 customer records:
After (duplicates identified):
name
selected
equivalence_class
A. Butoi
False
Alexandra Butoi
Alexandra Butoi
✓ True
Alexandra Butoi
Namoi Saphra
False
Naomi Saphra
Naomi Saphra
✓ True
Naomi Saphra
T. Gupta
False
Tejus Gupta
Tejus Gupta
✓ True
Tejus Gupta
The code:
from everyrow import create_client, create_session
from everyrow.ops import dedupe
import pandas as pd
async def find_duplicate_customers():
df = pd.read_csv("customers.csv")
async with create_client() as client:
async with create_session(client, name="Shopify Customer Dedupe") as session:
result = await dedupe(
session=session,
input=df,
equivalence_relation="""
Two rows are duplicates if they represent the same customer.
Consider:
- Name variations: J. Smith = John Smith, typos, nicknames
- Address variations: 123 Main St = 123 Main Street, Apt vs #
- Different emails for the same person
""",
)
return result.data
What it catches:
Name abbreviations: “A. Butoi” ↔ “Alexandra Butoi”
Typos: “Namoi Saphra” ↔ “Naomi Saphra”
Address variations: “123 Main St” ↔ “123 Main Street”
Different emails for same person
95% accuracy on “distractors” (same first name, different person)
Yeah, I have found this as well. And when I do export from Shopify to Excel/Sheets, only sometimes do the deduplication tools there work. Mostly they work only by exact match, but customer records can have different names, emails, etc.