Clean

Eliminate duplicate objects from JSON arrays based on full comparison or key properties

Remove duplicate records from JSON arrays based on designated key properties or full object equality.

🛡️ Your rows never leave your browser.
Why Onfyt: Performs semantic object hashing so identical objects with differing key ordering are accurately deduplicated.

Interactive Sample Data

idnameemailduplicate_status
101Grace Kerubograce@corp.comKEPT (first)
102Peter Njorogepeter@corp.comKEPT (first)
103Grace K.grace@corp.comREMOVED (duplicate email)

How this tool works

JSON arrays gathered from paginated webhooks or merged API exports frequently contain repeat objects that share the same customer ID or email address. These duplicates inflate database storage, trigger duplicate notification emails, and distort analytics dashboards built on row counts. Two objects that look identical can be serialized with their keys in different orders—{\"id\":1,\"name\":\"A\"} versus {\"name\":\"A\",\"id\":1}—causing string-based deduplication to miss the match entirely. Removing duplicates in JSON is more challenging than in flat CSV files because objects are associative: two objects with the exact same data might have their keys serialized in different orders ({'id': 1, 'name': 'A'} vs {'name': 'A', 'id': 1}). Naive string comparison will fail to recognize them as duplicates. DataFix performs semantic deduplication. You can deduplicate based on a designated key attribute (such as 'id' or 'email')—choosing whether to keep the first occurrence or the last occurrence—or perform full deep object equality that compares all properties regardless of key serialization sequence. JSON arrays often accumulate duplicate objects following distributed data collection or repeated API pagination calls. Preserving record order while deduplicating guarantees deterministic arrays for automated pipeline tests.

Things that usually go wrong

  • Deduplication by partial key: if you select 'email' only, two records with different names but the same email are deduplicated by email (the first record's name is kept).
  • Case differences in string keys are treated as distinct — normalise case first.

Edge cases we tested

🧪 Differing key order in identical objects ({'a': 1, 'b': 2} vs {'b': 2, 'a': 1})

Recognized as duplicates and deduplicated correctly.

🧪 Objects missing the deduplication key

Handled gracefully according to user preference (kept or grouped).

🧪 Case-insensitive key matching ('Grace@corp.com' vs 'grace@corp.com')

Normalized when case-insensitive deduplication is toggled on.

🧪 Two objects share id=1 but have different key counts ({note:"first"} on first only)

First occurrence kept: {id:1, name:"A", note:"first"}. Second object dropped even though it has fewer keys.

Frequently Asked Questions

Can I keep the last occurrence instead of the first?

Yes — include 'keep last' in your instruction (e.g. 'Deduplicate by id, keeping the last occurrence'). The Quick Chip 'Deduplicate keep-last by ID' pre-fills this for you.

Can I deduplicate on all fields?

Yes — leave the field selector empty to compare all fields for exact duplicate detection.

Related tools

Clean

Split Full Name into First and Last

Separate full name columns into dedicated first_name and last_name columns in CSV and Excel spreadsheets.

Open Tool →
Clean

Combine First and Last Name

Merge separate first and last name columns into a single clean full_name column, trimming extra spaces.

Open Tool →
Clean

Remove Duplicate Rows

Eliminate duplicate rows across your entire dataset or based on specific unique key columns.

Open Tool →