Golang memdb Tutorial [With Examples]


Written by - Tuan Nguyen
Reviewed by - Deepak Prasad

Introduction to golang memdb

In this tutorial, we will build a very simple key/value in-memory memdb database in Golang.

In-memory databases are purpose-built databases that store data primarily in memory, as opposed to databases that store data on disk or SSDs. In-memory data stores are intended to provide fast response times by eliminating the need for disk access. Because all data is stored and managed solely in main memory, in-memory databases are vulnerable to data loss in the event of a process or server failure. In-memory databases can save data to disk by logging each operation or taking snapshots.

In-memory databases are ideal for applications such as gaming leaderboards, session stores, and real-time analytics that require microsecond response times or have high traffic spikes.

Provides the memd package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consistency and Isolation from ACID. Being that it is in-memory, it does not provide durability. The database is instantiated with a schema that specifies the tables and indices that exist and allows transactions to be executed.

 

Install GO memdb package

You have to run the below command to install memdb package and its dependencies:

go get github.com/hashicorp/go-memdb

 

Perform CRUD Operation using GO memdb database

Create and Insert Records into database

Here is an example of creating a new schema and a new memdb database.

package main

import (
	"fmt"
	"github.com/hashicorp/go-memdb"
)

type Student struct {
	Id    int
	Name  string
	Score int
}

// Define the DB schema
func main() {
	schema := &memdb.DBSchema{
		Tables: map[string]*memdb.TableSchema{
			"student": &memdb.TableSchema{
				Name: "student",
				Indexes: map[string]*memdb.IndexSchema{
					"id": &memdb.IndexSchema{
						Name:    "id",
						Unique:  true,
						Indexer: &memdb.IntFieldIndex{Field: "Id"},
					},
					"name": &memdb.IndexSchema{
						Name:    "name",
						Unique:  false,
						Indexer: &memdb.StringFieldIndex{Field: "Name"},
					},
					"score": &memdb.IndexSchema{
						Name:    "score",
						Unique:  false,
						Indexer: &memdb.IntFieldIndex{Field: "Score"},
					},
				},
			},
		},
	}

	// Create a new data base
	db, err := memdb.NewMemDB(schema)
	if err != nil {
		panic(err)
	}

	// Create a write transaction
	txn := db.Txn(true)

	// Insert some people
	people := []*Student{
		&Student{1, "Anna", 95},
		&Student{2, "Bob", 75},
		&Student{3, "Dorothy", 80},
		&Student{4, "Daniel", 90},
	}

	// insert to db
	for _, p := range people {
		if err := txn.Insert("student", p); err != nil {
			panic(err)
		}
	}

	// Commit the transaction
	txn.Commit()
}

Explanation:

func (txn *Txn) Insert(table string, obj interface{}) error: Insert is used to add or update an object into the given table. When updating an object, the obj provided should be a copy rather than a value updated in-place. Modifying values in-place that are already inserted into MemDB is not supported behavior.

 

Query records in memdb database

To query in memdb, we can use several functions. For conditional query, we can use LowerBound() function:

func (txn *Txn) First(table, index string, args ...interface{}) (interface{}, error): First is used to return the first matching object for the given constraints on the index.

func (txn *Txn) Get(table, index string, args ...interface{}) (ResultIterator, error): Get is used to construct a ResultIterator over all the rows that match the given constraints of an index. The index values must match exactly (this is not a range-based or prefix-based lookup) by default. Prefix lookups: if the named index implements PrefixIndexer, you may perform prefix-based lookups by appending "_prefix" to the index name. In this scenario, the index values given in args are treated as prefix lookups. For example, a StringFieldIndex will match any string with the given value as a prefix: "mem" matches "memdb".

func (txn *Txn) Last(table, index string, args ...interface{}) (interface{}, error): Last is used to return the last matching object for the given constraints on the index.

func (txn *Txn) LowerBound(table, index string, args ...interface{}) (ResultIterator, error): LowerBound is used to construct a ResultIterator over all the the range of rows that have an index value greater than or equal to the provide args. Calling this then iterating until the rows are larger than required allows range scans within an index.

Let's consider the below code to understand how to use these functions, we use struct to store 1 record:

package main

import (
	"fmt"

	"github.com/hashicorp/go-memdb"
)

type Student struct {
	Id    int
	Name  string
	Score int
}

// Define the DB schema
func main() {
	schema := &memdb.DBSchema{
		Tables: map[string]*memdb.TableSchema{
			"student": &memdb.TableSchema{
				Name: "student",
				Indexes: map[string]*memdb.IndexSchema{
					"id": &memdb.IndexSchema{
						Name:    "id",
						Unique:  true,
						Indexer: &memdb.IntFieldIndex{Field: "Id"},
					},
					"name": &memdb.IndexSchema{
						Name:    "name",
						Unique:  false,
						Indexer: &memdb.StringFieldIndex{Field: "Name"},
					},
					"score": &memdb.IndexSchema{
						Name:    "score",
						Unique:  false,
						Indexer: &memdb.IntFieldIndex{Field: "Score"},
					},
				},
			},
		},
	}

	// Create a new data base
	db, err := memdb.NewMemDB(schema)
	if err != nil {
		panic(err)
	}

	// Create a write transaction
	txn := db.Txn(true)

	// Insert some people
	people := []*Student{
		&Student{1, "Anna", 95},
		&Student{2, "Bob", 75},
		&Student{3, "Dorothy", 80},
		&Student{4, "Daniel", 90},
	}

	// insert to db
	for _, p := range people {
		if err := txn.Insert("student", p); err != nil {
			panic(err)
		}
	}

	// Commit the transaction
	txn.Commit()

	// Create read-only transaction
	txn = db.Txn(false)
	defer txn.Abort()

	// query by name
	raw, err := txn.First("student", "name", "Anna")
	if err != nil {
		panic(err)
	}

	// Say hi!
	fmt.Printf("Hello student %s!\n", raw.(*Student).Name)

	// query all the people
	it, err := txn.Get("student", "name")
	if err != nil {
		panic(err)
	}

	fmt.Println("All the students:")
	for obj := it.Next(); obj != nil; obj = it.Next() {
		p := obj.(*Student)
		fmt.Printf("  %s\n", p.Name)
	}

	// Range scan over students with score between 75 and 90 inclusive
	it, err = txn.LowerBound("student", "score", 75)
	if err != nil {
		panic(err)
	}

	fmt.Println("Student with score 75 - 90:")
	for obj := it.Next(); obj != nil; obj = it.Next() {
		p := obj.(*Student)
		if p.Score > 90 {
			break
		}
		fmt.Printf("  %s with score %d\n", p.Name, p.Score)
	}
}

Output:

Hello student Anna!
All the students:
  Anna
  Bob
  Daniel
  Dorothy
Student with score 75 - 90:
  Bob with score 75
  Dorothy with score 80
  Daniel with score 90

 

Deleting records in memdb

We can use Delete() function  to remove 1 record, for multiple we will use deletion use DeleteAll() function.

func (txn *Txn) Delete(table string, obj interface{}) error: Delete is used to delete a single object from the given table. This object must already exist in the table

func (txn *Txn) DeleteAll(table, index string, args ...interface{}) (int, error): DeleteAll is used to delete all the objects in a given table matching the constraints on the index

Here is an example of deleting the record with Id=1, then delete all the students with 90 score.

package main

import (
	"fmt"

	"github.com/hashicorp/go-memdb"
)

// Create a struct
type Student struct {
	Id    int
	Name  string
	Score int
}

func main() {
	// Create the DB schema
	schema := &memdb.DBSchema{
		Tables: map[string]*memdb.TableSchema{
			"student": &memdb.TableSchema{
				Name: "student",
				Indexes: map[string]*memdb.IndexSchema{
					"id": &memdb.IndexSchema{
						Name:    "id",
						Unique:  true,
						Indexer: &memdb.IntFieldIndex{Field: "Id"},
					},
					"name": &memdb.IndexSchema{
						Name:    "name",
						Unique:  true,
						Indexer: &memdb.StringFieldIndex{Field: "Name"},
					},
					"score": &memdb.IndexSchema{
						Name:    "score",
						Unique:  false,
						Indexer: &memdb.IntFieldIndex{Field: "Score"},
					},
				},
			},
		},
	}

	// Create a new data base
	db, err := memdb.NewMemDB(schema)
	if err != nil {
		panic(err)
	}

	// Create a write transaction
	txn := db.Txn(true)

	// Insert some students
	people := []*Student{
		&Student{1, "Anna", 80},
		&Student{2, "Bob", 85},
		&Student{3, "Clair", 90},
		&Student{4, "Dorothy", 95},
		&Student{5, "Caitlyn", 90},
	}
	for _, p := range people {
		if err := txn.Insert("student", p); err != nil {
			panic(err)
		}
	}

	// Commit the transaction
	txn.Commit()

	printAll(db.Txn(false))

	fmt.Println("deleting one")

	txn = db.Txn(true)
	err = txn.Delete("student", Student{Id: 1})
	if err != nil {
		panic(err)
	}
	txn.Commit()
	printAll(db.Txn(false))

	fmt.Println("deleting many")
	txn = db.Txn(true)
	_, err = txn.DeleteAll("student", "score", 90)
	if err != nil {
		panic(err)
	}
	txn.Commit()
	printAll(db.Txn(false))
}

func printAll(txn *memdb.Txn) {
	// List all the students
	it, err := txn.Get("student", "id")
	if err != nil {
		panic(err)
	}

	fmt.Println("All the students:")
	for obj := it.Next(); obj != nil; obj = it.Next() {
		p := obj.(*Student)
		fmt.Printf("  %s\n", p.Name)
	}
}

Output:

All the students:
  Anna
  Bob
  Clair
  Dorothy
  Caitlyn
deleting one
All the students:
  Bob
  Clair
  Dorothy
  Caitlyn
deleting many
All the students:
  Bob
  Dorothy

 

Summary

In this article I have given some examples of working with memdb in golang. You can read more memdb functions in official documentation.

The database provides the following:

  • Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees the database is able to support any number of concurrent readers without locking, and allows a writer to make progress.
  • Transaction Support - The database allows for rich transactions, in which multiple objects are inserted, updated or deleted. The transactions can span multiple tables, and are applied atomically. The database provides atomicity and isolation in ACID terminology, such that until commit the updates are not visible.
  • Rich Indexing - Tables can support any number of indexes, which can be simple like a single field index, or more advanced compound field indexes. Certain types like UUID can be efficiently compressed from strings into byte indexes for reduced storage requirements.
  • Watches - Callers can populate a watch set as part of a query, which can be used to detect when a modification has been made to the database which affects the query results. This lets callers easily watch for changes in the database in a very general way.

 

References

https://en.wikipedia.org/wiki/In-memory_database
https://github.com/hashicorp/go-memdb

 

Views: 15

Tuan Nguyen

He is proficient in Golang, Python, Java, MongoDB, Selenium, Spring Boot, Kubernetes, Scrapy, API development, Docker, Data Scraping, PrimeFaces, Linux, Data Structures, and Data Mining. With expertise spanning these technologies, he develops robust solutions and implements efficient data processing and management strategies across various projects and platforms. You can connect with him on LinkedIn.

Categories GO

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment