Golang Concat Slices - Remove Duplicates [SOLVED]

In the previous article, I have shown you some examples of merging two or more slices in Golang. Today's post will show us how to concat slices and remove duplicates (keep unique elements).

 

Example 1: Remove duplicates from a string slice

The idea is to create a new slice, if the string first appears, add it to the slice and indicate that no more add this string to this new slice by using a map. Here is a helper function that allows us to remove duplicates from a slice:

Advertisement
package main

import "fmt"

func removeDuplicateString(strSlice []string) []string {
	// map to store unique keys
	keys := make(map[string]bool)
	returnSlice := []string{}
	for _, item := range strSlice {
		if _, value := keys[item]; !value {
			keys[item] = true
			returnSlice = append(returnSlice, item)
		}
	}
	return returnSlice
}

func main() {
	cities := []string{"Mumbai", "Ha Noi", "KL", "Ahmedabad", "BangKok", "Bangalore", "KL", "Kolkata", "Mumbai", "Pune", "New York", "Ha Noi"}

	// remove duplicate records
	uniqueCities := removeDuplicateStr(cities)

	// compare before and after removing
	fmt.Println("Slice before: ", cities)
	fmt.Println("Slice after:", uniqueCities)
}

Output:

[Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
[Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]

 

Example 2: Remove duplicate from a slice using Go generic

With generics, you can declare and use functions or types that are written to work with any of a set of types provided by calling code. In example 1, if we want to remove duplicates from int slice, we have to write another helper function:

func removeDuplicateInt(intSlice []int) []int {
    allKeys := make(map[int]bool)
    list := []int{}
    for _, item := range intSlice {
        if _, value := allKeys[item]; !value {
            allKeys[item] = true
            list = append(list, item)
        }
    }
    return list
}

Instead of that, we can use generics to write only 1 helper function and can remove duplicates from many data types slices:

package main

import "fmt"

func removeDuplicate[T comparable](sliceList []T) []T {
	allKeys := make(map[T]bool)
	list := []T{}
	for _, item := range sliceList {
		if _, value := allKeys[item]; !value {
			allKeys[item] = true
			list = append(list, item)
		}
	}
	return list
}

func main() {
	cities := []string{"Mumbai", "Ha Noi", "KL", "Ahmedabad", "BangKok", "Bangalore", "KL", "Kolkata", "Mumbai", "Pune", "New York", "Ha Noi"}

	// remove duplicate records
	uniqueCities := removeDuplicate(cities)

	// compare before and after removing
	fmt.Println("Slice before: ", cities)
	fmt.Println("Slice after:", uniqueCities)
	fmt.Println("-----------------------------")

	numbers := []int{1, 2, 3, 2, 2, 4, 5, 9, 7, 7, 11}
	// remove duplicate records
	uniqueNumbers := removeDuplicate(numbers)
	fmt.Println("Slice before: ", numbers)
	fmt.Println("Slice after:", uniqueNumbers)
	fmt.Println("-----------------------------")

	floats := []float64{1.5, 2.5, 3.5, 7.5, 8.5, 13.5, 8.5, 6.5, 0.5, 11.5, 11.5}
	// remove duplicate records
	uniqueFloats := removeDuplicate(floats)
	fmt.Println("Slice before: ", floats)
	fmt.Println("Slice after:", uniqueFloats)
	fmt.Println("-----------------------------")
}

Output:

Slice before:  [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
Slice after: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]
-----------------------------
Slice before:  [1 2 3 2 2 4 5 9 7 7 11]
Slice after: [1 2 3 4 5 9 7 11]
PS C:\Users\admin\Desktop\golang\src\main\chap2> go run .\slicedup.go
Slice before:  [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
Slice after: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]
-----------------------------
Slice before:  [1 2 3 2 2 4 5 9 7 7 11]
Slice after: [1 2 3 4 5 9 7 11]
-----------------------------
Slice before:  [1.5 2.5 3.5 7.5 8.5 13.5 8.5 6.5 0.5 11.5 11.5]
Slice after: [1.5 2.5 3.5 7.5 8.5 13.5 6.5 0.5 11.5]
-----------------------------

 

Example 3: Merge slices into 1 slice and then remove duplicates

We can merge slices into 1 slice and then use the helper function in example 2 to remove duplicates. The below example shows you how to append multiple slices and then eliminate duplicate elements:

package main

import "fmt"

func removeDuplicate[T comparable](sliceList []T) []T {
	allKeys := make(map[T]bool)
	list := []T{}
	for _, item := range sliceList {
		if _, value := allKeys[item]; !value {
			allKeys[item] = true
			list = append(list, item)
		}
	}
	return list
}

func main() {
	intSlice := []int{2, 4, 10, 6}
	// append all slices into one
	intSlice = append(intSlice, []int{8, 11, 10}...)
	intSliceNew := []int{1, 3, 5, 2, 8}
	intSlice = append(intSlice, intSliceNew...)
	fmt.Println("Slice after 2 append:", intSlice)

        // remove duplicates
	uniqueNumbers := removeDuplicate(intSlice)
	fmt.Println("Slice after remove duplicates:", uniqueNumbers)
}

Output:

Advertisement
Slice after 2 append: [2 4 10 6 8 11 10 1 3 5 2 8]
Slice after remove duplicates: [2 4 10 6 8 11 1 3 5]

 

Example 4: Using a loop to iterate through all slices and remove duplicates

In this example, we will iterate all the slices within a loop and create a new slice to store the unique elements:

package main

import "fmt"

func main() {
	firstIntSlice := []int{1, 2, 3, 3, 3}
	secondIntSlice := []int{2, 3, 4, 5, 5, 6}
	thirdIntSlice := []int{7, 2, 10, 5, 6, 17}

	// remove duplicate from a slice
	fmt.Println("Slice 2 before remove duplicates:", secondIntSlice)
	fmt.Println("Slice 2 after remove duplicates:", UniqueSlice(secondIntSlice))

	// remove duplicate from multiple merged slices
	fmt.Println(UniqueSlice(firstIntSlice, secondIntSlice, thirdIntSlice))

}

func UniqueSlice[T comparable](dataSlices ...[]T) []T {
	uniqueMap := map[T]bool{}

	for _, dataSlice := range dataSlices {
		for _, number := range dataSlice {
			uniqueMap[number] = true
		}
	}

	// Slice with the fixed capacity of unique items to optimize memory
	result := make([]T, 0, len(uniqueMap))

	for key := range uniqueMap {
		result = append(result, key)
	}

	return result
}

Output:

Slice 2 before remove duplicates: [2 3 4 5 5 6]
Slice 2 after remove duplicates: [6 2 3 4 5]
[1 2 5 6 3 4 7 10 17]

 

Summary

In today's post, I have given examples of removing duplicates in a slice as well as merged slices. With the help of go generics, we can easily write 1 function and use it for many different slice data types. After this article, you can understand how to use map to store unique keys and go generics to shorten your code.

 

References

https://go.dev/doc/tutorial/generics

 

Categories GO

Didn't find what you were looking for? Perform a quick search across GoLinuxCloud

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can either use the comments section or contact me form.

Thank You for your support!!

Leave a Comment

X