In the previous article, I have shown you some examples of merging two or more slices in Golang. Today's post will show us how to concat slices and remove duplicates (keep unique elements).
Example 1: Remove duplicates from a string slice
The idea is to create a new slice, if the string first appears, add it to the slice and indicate that no more add this string to this new slice by using a map. Here is a helper function that allows us to remove duplicates from a slice:
package main
import "fmt"
func removeDuplicateString(strSlice []string) []string {
// map to store unique keys
keys := make(map[string]bool)
returnSlice := []string{}
for _, item := range strSlice {
if _, value := keys[item]; !value {
keys[item] = true
returnSlice = append(returnSlice, item)
}
}
return returnSlice
}
func main() {
cities := []string{"Mumbai", "Ha Noi", "KL", "Ahmedabad", "BangKok", "Bangalore", "KL", "Kolkata", "Mumbai", "Pune", "New York", "Ha Noi"}
// remove duplicate records
uniqueCities := removeDuplicateStr(cities)
// compare before and after removing
fmt.Println("Slice before: ", cities)
fmt.Println("Slice after:", uniqueCities)
}
Output:
[Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
[Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]
Example 2: Remove duplicate from a slice using Go generic
With generics, you can declare and use functions or types that are written to work with any of a set of types provided by calling code. In example 1, if we want to remove duplicates from int slice, we have to write another helper function:
func removeDuplicateInt(intSlice []int) []int {
allKeys := make(map[int]bool)
list := []int{}
for _, item := range intSlice {
if _, value := allKeys[item]; !value {
allKeys[item] = true
list = append(list, item)
}
}
return list
}Instead of that, we can use generics to write only 1 helper function and can remove duplicates from many data types slices:
package main
import "fmt"
func removeDuplicate[T comparable](sliceList []T) []T {
allKeys := make(map[T]bool)
list := []T{}
for _, item := range sliceList {
if _, value := allKeys[item]; !value {
allKeys[item] = true
list = append(list, item)
}
}
return list
}
func main() {
cities := []string{"Mumbai", "Ha Noi", "KL", "Ahmedabad", "BangKok", "Bangalore", "KL", "Kolkata", "Mumbai", "Pune", "New York", "Ha Noi"}
// remove duplicate records
uniqueCities := removeDuplicate(cities)
// compare before and after removing
fmt.Println("Slice before: ", cities)
fmt.Println("Slice after:", uniqueCities)
fmt.Println("-----------------------------")
numbers := []int{1, 2, 3, 2, 2, 4, 5, 9, 7, 7, 11}
// remove duplicate records
uniqueNumbers := removeDuplicate(numbers)
fmt.Println("Slice before: ", numbers)
fmt.Println("Slice after:", uniqueNumbers)
fmt.Println("-----------------------------")
floats := []float64{1.5, 2.5, 3.5, 7.5, 8.5, 13.5, 8.5, 6.5, 0.5, 11.5, 11.5}
// remove duplicate records
uniqueFloats := removeDuplicate(floats)
fmt.Println("Slice before: ", floats)
fmt.Println("Slice after:", uniqueFloats)
fmt.Println("-----------------------------")
}Output:
Slice before: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
Slice after: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]
-----------------------------
Slice before: [1 2 3 2 2 4 5 9 7 7 11]
Slice after: [1 2 3 4 5 9 7 11]
PS C:\Users\admin\Desktop\golang\src\main\chap2> go run .\slicedup.go
Slice before: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore KL Kolkata Mumbai Pune New York Ha Noi]
Slice after: [Mumbai Ha Noi KL Ahmedabad BangKok Bangalore Kolkata Pune New York]
-----------------------------
Slice before: [1 2 3 2 2 4 5 9 7 7 11]
Slice after: [1 2 3 4 5 9 7 11]
-----------------------------
Slice before: [1.5 2.5 3.5 7.5 8.5 13.5 8.5 6.5 0.5 11.5 11.5]
Slice after: [1.5 2.5 3.5 7.5 8.5 13.5 6.5 0.5 11.5]
-----------------------------
Example 3: Merge slices into 1 slice and then remove duplicates
We can merge slices into 1 slice and then use the helper function in example 2 to remove duplicates. The below example shows you how to append multiple slices and then eliminate duplicate elements:
package main
import "fmt"
func removeDuplicate[T comparable](sliceList []T) []T {
allKeys := make(map[T]bool)
list := []T{}
for _, item := range sliceList {
if _, value := allKeys[item]; !value {
allKeys[item] = true
list = append(list, item)
}
}
return list
}
func main() {
intSlice := []int{2, 4, 10, 6}
// append all slices into one
intSlice = append(intSlice, []int{8, 11, 10}...)
intSliceNew := []int{1, 3, 5, 2, 8}
intSlice = append(intSlice, intSliceNew...)
fmt.Println("Slice after 2 append:", intSlice)
// remove duplicates
uniqueNumbers := removeDuplicate(intSlice)
fmt.Println("Slice after remove duplicates:", uniqueNumbers)
}Output:
Slice after 2 append: [2 4 10 6 8 11 10 1 3 5 2 8]
Slice after remove duplicates: [2 4 10 6 8 11 1 3 5]
Example 4: Using a loop to iterate through all slices and remove duplicates
In this example, we will iterate all the slices within a loop and create a new slice to store the unique elements:
package main
import "fmt"
func main() {
firstIntSlice := []int{1, 2, 3, 3, 3}
secondIntSlice := []int{2, 3, 4, 5, 5, 6}
thirdIntSlice := []int{7, 2, 10, 5, 6, 17}
// remove duplicate from a slice
fmt.Println("Slice 2 before remove duplicates:", secondIntSlice)
fmt.Println("Slice 2 after remove duplicates:", UniqueSlice(secondIntSlice))
// remove duplicate from multiple merged slices
fmt.Println(UniqueSlice(firstIntSlice, secondIntSlice, thirdIntSlice))
}
func UniqueSlice[T comparable](dataSlices ...[]T) []T {
uniqueMap := map[T]bool{}
for _, dataSlice := range dataSlices {
for _, number := range dataSlice {
uniqueMap[number] = true
}
}
// Slice with the fixed capacity of unique items to optimize memory
result := make([]T, 0, len(uniqueMap))
for key := range uniqueMap {
result = append(result, key)
}
return result
}Output:
Slice 2 before remove duplicates: [2 3 4 5 5 6]
Slice 2 after remove duplicates: [6 2 3 4 5]
[1 2 5 6 3 4 7 10 17]
Summary
In today's post, I have given examples of removing duplicates in a slice as well as merged slices. With the help of go generics, we can easily write 1 function and use it for many different slice data types. After this article, you can understand how to use map to store unique keys and go generics to shorten your code.
References
https://go.dev/doc/tutorial/generics
