How to Remove Duplicate Dictionaries in a List

The Challenge

Create a function to remove duplicate dictionaries from a list. A dictionary is considered a duplicate if it has the exact same keys and values. The function should accommodate deep comparisons (nested dictionaries).

The Solution

Use list comprehension and the json library to sort and stringify each dictionary and the set data type to ensure unique values.import jsondef remove_dupe_dicts(l):
 list_of_strings = [
   json.dumps(d, sort_keys=True)
   for d in l
 ]list_of_strings = set(list_of_strings)return [
   json.loads(s)
   for s in list_of_strings
 ]

Testing our function, we pass in some sample data that include nested dictionaries with similar and dissimilar nested keys.data = [
 {"id": 1, "sub": {"id": 1}},
 {"id": 1, "sub": {"id": 2}},
 {"id": 4},
 {"id": 1, "sub": {"id": 1}}
]print(remove_dupe_dicts(data))
"""
[
 {'id': 1, 'sub': {'id': 1}},
 {'id': 4},
 {'id': 1, 'sub': {'id': 2}}
]
"""

Honorable Mentions

Convert Dictionaries to Tuples: If you do not want to use the json library because your dictionary is not valid JSON then converting the list of dictionaries to a list of tuples is a viable option. Keep in mind the coercion will not work with nested data structures.

def remove_dupe_dicts(l): 
  return [ 
    dict(t)  
    for t in { 
      tuple(d.items()) 
      for d in l 
    } 
  ]