body {
font-family: sans-serif;
}
h1, h2, p, ul, li {
margin: 1em;
}
pre {
background-color: #f0f0f0;
padding: 1em;
border: 1px solid #ccc;
overflow-x: auto;
}
code {
font-family: monospace;
}

Python Advanced: Remove Duplicates while Preserving Order

This tutorial demonstrates how to remove duplicate elements from a list in Python while maintaining the original order of the elements. This is a common task when dealing with data cleaning and preprocessing.

Method 1: Using a Loop and a New List

The simplest approach involves iterating through the original list and adding elements to a new list only if they haven’t been encountered before.

Step 1: Define the function

def remove_duplicates_loop(input_list):
  """Removes duplicates from a list while preserving order using a loop."""
  seen = set() # Use a set for efficient duplicate checking
  result = []
  for item in input_list:
    if item not in seen:
      seen.add(item)
      result.append(item)
  return result

Step 2: Call the function

my_list = [1, 2, 2, 3, 4, 4, 5, 1]
unique_list = remove_duplicates_loop(my_list)
print(unique_list)

Step 3: Explanation

First, we initialize an empty set called `seen`. Sets in Python provide very fast lookups (checking if an element is present). This significantly improves performance compared to using a list for duplicate checking, especially for large lists. We also initialize an empty list called `result` which will store the unique elements in their original order.

The code iterates through each `item` in the `input_list`. In each iteration, it checks if `item` is already present in the `seen` set. If it isn’t, it means we haven’t encountered this element before. We add `item` to the `seen` set and then append it to the `result` list.

Finally, the function returns the `result` list containing the unique elements in the original order.

#Example Usage:
# Input: [1, 2, 2, 3, 4, 4, 5, 1]
#
# Loop 1: item = 1. 1 not in seen. Add 1 to seen and result. seen = {1}, result = [1]
# Loop 2: item = 2. 2 not in seen. Add 2 to seen and result. seen = {1, 2}, result = [1, 2]
# Loop 3: item = 2. 2 in seen. Do nothing.
# Loop 4: item = 3. 3 not in seen. Add 3 to seen and result. seen = {1, 2, 3}, result = [1, 2, 3]
# Loop 5: item = 4. 4 not in seen. Add 4 to seen and result. seen = {1, 2, 3, 4}, result = [1, 2, 3, 4]
# Loop 6: item = 4. 4 in seen. Do nothing.
# Loop 7: item = 5. 5 not in seen. Add 5 to seen and result. seen = {1, 2, 3, 4, 5}, result = [1, 2, 3, 4, 5]
# Loop 8: item = 1. 1 in seen. Do nothing.
# Output: [1, 2, 3, 4, 5]

Output:

[1, 2, 3, 4, 5]

Method 2: Using `dict.fromkeys()`

Python dictionaries inherently store only unique keys. We can leverage this to remove duplicates while preserving order (as of Python 3.7, dictionaries preserve insertion order).

Step 1: Define the function

def remove_duplicates_dict(input_list):
  """Removes duplicates from a list while preserving order using dict.fromkeys()."""
  return list(dict.fromkeys(input_list))

Step 2: Call the function

my_list = [1, 2, 2, 3, 4, 4, 5, 1]
unique_list = remove_duplicates_dict(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5]

Step 3: Explanation

This method is concise and efficient. `dict.fromkeys(input_list)` creates a dictionary where the elements of `input_list` are used as keys. Since dictionary keys must be unique, any duplicates are automatically discarded. The `list()` function then converts the dictionary’s keys back into a list, preserving the order in which the keys were first inserted (which is the original order of the elements in the input list).

Method 3: Using List Comprehension (Advanced)

List comprehension provides a compact way to create new lists based on existing iterables. We can combine this with a set for efficient duplicate detection.

Step 1: Define the function

def remove_duplicates_comprehension(input_list):
  """Removes duplicates from a list while preserving order using list comprehension."""
  seen = set()
  return [x for x in input_list if not (x in seen or seen.add(x))]

Step 2: Call the function

my_list = [1, 2, 2, 3, 4, 4, 5, 1]
unique_list = remove_duplicates_comprehension(my_list)
print(unique_list)

Output:

[1, 2, 3, 4, 5]

Step 3: Explanation

This method is more compact but can be harder to read for beginners. It utilizes a list comprehension to iterate through the `input_list`. For each element `x`, it checks if `x` is in the `seen` set. If `x` is not in `seen`, it means it’s the first time we’ve encountered it. The `seen.add(x)` part adds `x` to the `seen` set. The `or` operator ensures that `seen.add(x)` is always executed, which is necessary for the set to be updated correctly within the list comprehension’s scope. The condition `not (x in seen or seen.add(x))` effectively includes the element in the new list only if it’s not already in `seen`. The final list comprehension constructs a new list with the unique elements.