Python

Remove Duplicates From a Python List

The list data structure is quite popular among the Python programmers and users. Sometimes, the users encounter duplicate values in the list and find themselves unable to remove them. If so, then this guide is meant for you. Within this guide, we will be discussing several methods to remove duplicate values from a list data structure. Let’s have a fresh look.

Method 01: Naïve

Start by opening the terminal shell with “Ctrl+Alt+T”. Make a new Python file with the “touch” instruction. The file “dup.py” will be opened using the “GNU Nano” editor command:

$ touch dup.py

$ Nano dup.py

After opening the file in Nano editor, add the Python support at the start of it. The integer type list has been initialized, with some of its values being repeated. The print statement has been displaying the list by converting its value to a string type. Another empty list, “Arr”, has been initialized. The “for” loop will be used to simply put the first occurrence of a list value and add it to the new list “Arr” using the “append” function. The newly made list will be printed out:

    #! /usr/bin/python3

    List = [1, 4, 6, 7, 10, 2, 1, 4, 3]

    Arr = []

    For i in List

    If I not in Arr :

    Arr.append()

    Print(“ The list after update: “ + str(Arr))

Upon running the Python file, the terminal has been displayed by the original list and the new one with no duplicates in it:

$ python3 dup.py

Method 02: List Comprehension

The list comprehension method is quite similar to the naïve method. The only change is the position of using the append method, “for” loop and “if” statement in it. Open the same file “dup.py” to update its code. The list comprehension method is shown as the mix of “for” loop and “if” statement with the append() function. The rest of the code is the same:

    #! /usr/bin/python3

    List = [1, 4, 6, 7, 10, 2, 1, 4, 3]

    Print(“ The list before update: “ + str(List))

    Arr = []

    [Arr.append() for i in List if i not in Arr ]:

    Print(“ The list after update: “ + str(Arr))

After executing the Python file, we have the expected result, i.e., the original list and updated the new list:

$ python3 dup.py

Method 03: Enumerate With List Comprehension

You can utilize the enumerate function within the list comprehension to avoid duplicates in the list. Simply open the file “dup.py”, and add the enumerate method within the list comprehension line between the “for” loop. Both the old list and the updated list with no duplicates will be displayed on the shell with the use of the “print” statement in the code:

    #! /usr/bin/python3

    List = [“L”, “I”, “n”, “u”, “x”, “L”, “n”]

    Print(“ The list before update: “ + str(List))

    Arr = [I for n, I in enumerate(List) if I not in List[:n]]

    Print(“ The list after update: “ + str(Arr))

The file has been executed, and the result has displayed the original list first, then the updated new list that has no duplicate values. The enumerate function also ordered the list:

$ python3 dup.py

Example 04: Set() Function

The set() function is quite known to remove duplicates from the list. It removes the duplicates but changes the order of a list on its own. So, open the file dup.py and add the list initialization. The original list has been printed out. Then, we have used the “set()” method on “List” to remove duplicates and again convert the remaining ones into a list. The resultant list has been updated and displayed on the shell again:

    #! /usr/bin/python3

    List = [“L”, “I”, “n”, “u”, “x”, “L”, “n”]

    Print(“ The list before update: “ + str(List))

    List = list(set(List))

    Print(“ The list after update: “ + str(List))

Upon running the Python file, the original list and then, the updated one has been displayed.

$ python3 dup.py

Method 05: Collections.OrderedDict

The module collections can also be used for this purpose. Simply import the OrderedDict package from the “collections” library. Initialize the list and print it. A new variable, “Arr”, is used to get the list generated by the “Ordered.Dict.fromkeys()” method and bypassing the original list in it. The newly created list variable will be displayed:

    #! /usr/bin/python3

    From collections import OrderedDict

    List = [“L”, “I”, “n”, “u”, “x”, “L”, “n”]

    Print(“ The list before update: “ + str(List))

    Arr = list(OrderedDict.fromkeys(List))

    Print(“ The list after update: “ + str(Arr))

The execution shows the expected result. The first list is the original one, and the second one is the updated list with no duplicates in it:

$ python3 dup.py

Conclusion:

This article presented several methods to remove duplicates from the list, i.e., list comprehension, set() functions, enumerate, collections.OrderedDict package import. We hope this guide will help you at its best, and please check out Linux Hint for more informative articles.

About the author

Kalsoom Bibi

Hello, I am a freelance writer and usually write for Linux and other technology related content