Fuzzy-Match Product Exclusion Classification — Test Results
Fuzzy-Match Product Exclusion Classification — Test Results
Section titled “Fuzzy-Match Product Exclusion Classification — Test Results”What We Built
Section titled “What We Built”A classifier that determines whether a product should be excluded from a Button merchant’s enrichment based on the merchant’s exclusion terms (e.g., Walmart excludes “Grocery, Food & Beverages”, “Beauty”, “Pharmacy”). Today these terms are displayed as T&C text but never enforced programmatically.
The classifier uses fuzzy string matching (RapidFuzz WRatio) to compare product attributes (category, name, brand) against a merchant’s exclusion list. It’s designed to err on the side of letting products through — we’d rather show a product that should be excluded than block one that shouldn’t be.
How It Works
Section titled “How It Works”Product: Dove Body Wash Category: "PERSONAL CARE" ──┐ Name: "Dove Body Wash..." ──┼── fuzzy match against ──> Walmart exclusions: Brand: "DOVE" ──┘ "Grocery, Food & Beverages" "Personal Care (Bath, Body, Oral & Feminine)" ← match (90.0) "Beauty" "Baby (Bedding, Bags, Diapers & Hardlines)" ...
Result: EXCLUDED (category matched "Personal Care" at score 90/100)Matching pipeline:
- Extract top-level product category (before first
|or,delimiter) - Strip domain-specific stopwords from both product attributes and exclusion terms (“all”, “select”, “products”, “orders”, “over”, etc.)
- Skip queries shorter than 4 characters (too noisy)
- Send category, name, and brand to fuzzy matcher — first match wins (category checked first for highest precision)
- Threshold: 80/100 similarity score (tuned via testing)
- Results cached to disk with 30-day TTL
Key design decisions:
- Category is checked first because it has the highest precision (98%+)
- Errors in the fuzzy match sidecar return “not excluded” (non-blocking)
Test Setup
Section titled “Test Setup”- 10,000 products pulled from production Snowflake (FIDO catalog — real products across all categories)
- 55 merchants from production Button service (with non-gift-card exclusions and a retailer ID)
- 550,000 cross-tests (every product against every merchant)
- Completed in 6.9 seconds
Results
Section titled “Results”| Metric | Value |
|---|---|
| Total exclusions | 11,388 / 550,000 (2.07%) |
| Match by category | 7,402 (65%) |
| Match by product name | 3,614 (32%) |
| Match by brand | 372 (3%) |
Per-Merchant Exclusion Rates (selected)
Section titled “Per-Merchant Exclusion Rates (selected)”| Merchant | Excluded / 10k | Exclusion Terms |
|---|---|---|
| Walmart | 2,148 (21.5%) | 12 terms covering grocery, personal care, beauty, baby, pets, pharmacy, auto, video games |
| JCPenney | 1,052 (10.5%) | Furniture, Kitchen, Patio, Mattresses, TVs, Game consoles, Electronics, Appliances |
| Gap | 981 (9.8%) | Brand collaboration exclusions (Gap x DOEN, Gap x Palace, Gap x Love Shack Fancy) |
| Disney+ | 435 (4.4%) | Free trials, Premium Duo Bundle |
| eBay | 375 (3.8%) | Household cleaners, paper towels, travel, hand sanitizers, heavy equipment |
| Sam’s Club | 305 (3.1%) | Grocery & Food, Alcohol, Cough/Cold medicine, Restaurant Supplies |
| Walgreens | 297 (3.0%) | Grocery, Household products, Electronics, Diet & Fitness |
| Best Buy | 260 (2.6%) | Laptops/Tablets, Video Game Hardware, All Apple Products, POSA cards |
| Kohl’s | 176 (1.8%) | Kohl’s Cares, Apple/Amazon/Nintendo/Google products, Gaming, Crocs, Jewelry over $2k |
| Nike | 57 (0.6%) | Gift cards, Air Jordans, Converse, Hyperice |
| CVS | 36 (0.4%) | Gift cards, Discounts, COVID tests |
Merchants with 0 exclusions: Athleta, Belk, Children’s Place, Crocs (only “Limited edition collaborations”), e.l.f., Fanatics, JD Sports, Old Navy, PacSun, StockX, Temu, Zappos
What’s Working Well
Section titled “What’s Working Well”Category matching (65% of exclusions) is highly accurate. These are clear, correct matches:
| Product Category | Exclusion Term | Merchant |
|---|---|---|
| GROCERY | Grocery, Food & Beverages | Walmart |
| BEAUTY | Beauty | Walmart |
| DIAPERS | Baby (Bedding, Bags, Diapers & Hardlines) | Walmart, ACE |
| WINE | Wine | Naked Wines |
| AUTOMOTIVE | Automotive Tires | Walmart |
| FURNITURE | Furniture | JCPenney |
| ELECTRONICS | Electronics | Walgreens, Verizon |
| PET FOOD | Grocery, Food & Beverages | Home Depot |
Exact brand matches work perfectly:
| Brand | Exclusion | Merchant | Score |
|---|---|---|---|
| BISSELL | Bissell | Kohl’s | 100 |
| CONVERSE | Converse | Nike | 100 |
| CROCS | Crocs | Kohl’s | 100 |
| REEBOK | Reebok items | Journeys | 100 |
| NIKE | Select Nike items | Finish Line | 100 |
| HP | Select HP products | Staples | 100 |
| APPLE | All Apple-Branded Products | Best Buy | 90 |
Product name matching catches real exclusions that category alone misses:
| Product Name | Exclusion | Merchant |
|---|---|---|
| ”Charmin Ultra Strong Toilet Tissue…” | Paper towels and toilet paper | eBay |
| ”Bounty Select-a-size Paper Towel…” | Paper towels and toilet paper | eBay |
| ”Miami Baked Challah Bread” | Bread | Aldi |
| ”Huggies Snug & Dry Baby Diapers…” | Baby (Bedding, Bags, Diapers) | Walmart |
Known Risks & Limitations
Section titled “Known Risks & Limitations”Risk 1: Channel exclusions misread as product exclusions (medium impact)
Section titled “Risk 1: Channel exclusions misread as product exclusions (medium impact)”~10% of exclusion terms describe how/where you buy, not what you buy. The fuzzy matcher can’t distinguish these:
| Exclusion Term | Merchant | Problem |
|---|---|---|
| ”Purchases made on BoohooMan (Men’s Clothing)“ | Boohoo | Matches all men’s clothing products (3,331 false blocks). This exclusion means purchases on the BoohooMan website — not that men’s clothing is excluded. |
| ”Scan and Go Orders” | Sam’s Club | Matches product names containing “Go" |
| "Orders placed on mobile web” | Uber | Matches product names containing “web" |
| "Bookings in Russia or Ukraine” | IHG | Not a product category at all |
| ”Disney Premium Duo Bundle Upgrade” | Disney+ | Matches Disney-branded products |
Mitigation: Curate a denylist of channel-type exclusions to skip before fuzzy matching, or add a channel_exclusion flag to the Button data model.
Risk 2: Fuzzy name matching has ~5-10% false positive rate
Section titled “Risk 2: Fuzzy name matching has ~5-10% false positive rate”Product names are long, natural-language strings. Some false matches are unavoidable:
| Product Name | Falsely Matched To | Score | Why |
|---|---|---|---|
| ”Crestlive Products 10 Ft Aluminum…" | "Apple, Amazon, Nintendo, or Google products” | 85 | Word “Products” (now mitigated by stopword) |
| “BALLOON DELIGHTS" | "Flights” (Expedia) | 83 | Suffix “-LIGHTS” ≈ “Flights" |
| "PAMPERS" | "Paper” (HP) | 83 | PAP substring |
| ”GARMIN" | "Gaming” (Kohl’s) | 83 | GAMI substring |
Mitigation: These are conservative — a product falsely excluded at one merchant still enriches normally at other merchants. The 80/100 threshold was tuned to minimize these. Category matching alone (65% of all matches) has <2% FP rate.
Risk 3: Short or generic exclusion terms match too broadly
Section titled “Risk 3: Short or generic exclusion terms match too broadly”| Exclusion | Merchant | Issue |
|---|---|---|
| ”All non-grocery items” | Target | After stopword stripping, becomes “non-grocery” — better, but still vague |
| ”Select SKUs” | Athleta, Belk, Fanatics, HSN, etc. | Eliminated by stopword stripping — but these merchants get 0 exclusions now, which may or may not be correct |
| ”Discounts” | 8 merchants | Matches discount-related product names |
Guards We Added
Section titled “Guards We Added”| Guard | What It Does | Impact |
|---|---|---|
| Top-level category only | Strips sub-categories (e.g. “ORAL CARE” from “ORAL CARE|TOOTHPASTE|ADULT”) | Eliminated sub-category FPs like “BEEF” matching “Beer”, “AB” matching “JLab or Beats” |
| Exclusion stopwords | Strips “all”, “select”, “products”, “orders”, “over”, etc. from both sides before matching | ”Select SKUs” → 0 FPs, “All non-grocery” 84 → 2, “Apple/Amazon/Nintendo/Google products” → 0 |
| Min query length (4 chars) | Skips brands/names shorter than 4 characters | Eliminated 122 noisy short brands (ON, SO, GE, HP) from matching |
| Threshold 80/100 | Requires high similarity score | Prevents weak partial matches while allowing category-level matches |