How To Flatten a Dictionary With Nested Lists and Dictionaries in Python
Learn to flatten a dictionary with a custom separator, accommodating nested lists and dictionaries
Have you ever wondered how to get your multi-layered data structures into a tabular format? Here is a function that will flatten a dictionary, which accommodates nested lists and dictionaries.
Tuples and other data types are not included because this is primarily used when manipulating JSON data. The function allows for a custom separator and will preserve the order of the terms.
Starting Structure
Our function will require recursion to properly function. Recursion is the process of invoking one’s self.
Additionally, we will need to use the collections library to create an OrderedDict to preserve the order of terms. Here’s the skeleton of our function:
Notice the inclusion of a sep parameter that will default to an underscore if no custom separator is specified.
The obj variable is used to build our flattened dictionary and will be added to at the end of each recursion.
Inside our recurse() function, the parent_key variable will be used to pass the current key, allowing for concatenation of the previous key and new key(s).
Outlining the recurse() Function
Since we’re sticking to JSON compliant data, there are three possible types of data: list/array, dictionary/object, and primitive values.
Setting up these three scenarios, the recurse() function takes initial shape.
Notice on line 11, we write to obj. The only time we will be writing to our flattened object is when the data type is a primitive—not a list or dictionary.
Completing the flatten() Function
Fleshing out the remainder of the function, the parent_key for lists and dictionaries varies because the index is required for lists. If parent_key is empty, then only the current index or current key will be passed.
Testing Out the Function
Here’s some sample data that includes nested lists, dictionaries, and values.
When calling our flatten() function using the above data, here is the result using pprint() to cleanly format the output:
Notice that nested data structures are identified as the original key, concatenated via a separator with the new key. For nested lists, the index is also included as well.
For the full function and test execution in one place, check out this gist.