Table of Contents
Introduction to golang memdb
In this tutorial, we will build a very simple key/value in-memory memdb database in Golang.
In-memory databases are purpose-built databases that store data primarily in memory, as opposed to databases that store data on disk or SSDs. In-memory data stores are intended to provide fast response times by eliminating the need for disk access. Because all data is stored and managed solely in main memory, in-memory databases are vulnerable to data loss in the event of a process or server failure. In-memory databases can save data to disk by logging each operation or taking snapshots.
In-memory databases are ideal for applications such as gaming leaderboards, session stores, and real-time analytics that require microsecond response times or have high traffic spikes.
Provides the memd package that implements a simple in-memory database built on immutable radix trees. The database provides Atomicity, Consistency and Isolation from ACID. Being that it is in-memory, it does not provide durability. The database is instantiated with a schema that specifies the tables and indices that exist and allows transactions to be executed.
Install GO memdb package
You have to run the below command to install memdb package and its dependencies:
go get github.com/hashicorp/go-memdb
Perform CRUD Operation using GO memdb database
Create and Insert Records into database
Here is an example of creating a new schema and a new memdb database.
package main
import (
"fmt"
"github.com/hashicorp/go-memdb"
)
type Student struct {
Id int
Name string
Score int
}
// Define the DB schema
func main() {
schema := &memdb.DBSchema{
Tables: map[string]*memdb.TableSchema{
"student": &memdb.TableSchema{
Name: "student",
Indexes: map[string]*memdb.IndexSchema{
"id": &memdb.IndexSchema{
Name: "id",
Unique: true,
Indexer: &memdb.IntFieldIndex{Field: "Id"},
},
"name": &memdb.IndexSchema{
Name: "name",
Unique: false,
Indexer: &memdb.StringFieldIndex{Field: "Name"},
},
"score": &memdb.IndexSchema{
Name: "score",
Unique: false,
Indexer: &memdb.IntFieldIndex{Field: "Score"},
},
},
},
},
}
// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
panic(err)
}
// Create a write transaction
txn := db.Txn(true)
// Insert some people
people := []*Student{
&Student{1, "Anna", 95},
&Student{2, "Bob", 75},
&Student{3, "Dorothy", 80},
&Student{4, "Daniel", 90},
}
// insert to db
for _, p := range people {
if err := txn.Insert("student", p); err != nil {
panic(err)
}
}
// Commit the transaction
txn.Commit()
}Explanation:
func (txn *Txn) Insert(table string, obj interface{}) error: Insert is used to add or update an object into the given table. When updating an object, the obj provided should be a copy rather than a value updated in-place. Modifying values in-place that are already inserted into MemDB is not supported behavior.
Query records in memdb database
To query in memdb, we can use several functions. For conditional query, we can use LowerBound() function:
func (txn *Txn) First(table, index string, args ...interface{}) (interface{}, error): First is used to return the first matching object for the given constraints on the index.
func (txn *Txn) Get(table, index string, args ...interface{}) (ResultIterator, error): Get is used to construct a ResultIterator over all the rows that match the given constraints of an index. The index values must match exactly (this is not a range-based or prefix-based lookup) by default. Prefix lookups: if the named index implements PrefixIndexer, you may perform prefix-based lookups by appending "_prefix" to the index name. In this scenario, the index values given in args are treated as prefix lookups. For example, a StringFieldIndex will match any string with the given value as a prefix: "mem" matches "memdb".
func (txn *Txn) Last(table, index string, args ...interface{}) (interface{}, error): Last is used to return the last matching object for the given constraints on the index.
func (txn *Txn) LowerBound(table, index string, args ...interface{}) (ResultIterator, error): LowerBound is used to construct a ResultIterator over all the the range of rows that have an index value greater than or equal to the provide args. Calling this then iterating until the rows are larger than required allows range scans within an index.
Let's consider the below code to understand how to use these functions, we use struct to store 1 record:
package main
import (
"fmt"
"github.com/hashicorp/go-memdb"
)
type Student struct {
Id int
Name string
Score int
}
// Define the DB schema
func main() {
schema := &memdb.DBSchema{
Tables: map[string]*memdb.TableSchema{
"student": &memdb.TableSchema{
Name: "student",
Indexes: map[string]*memdb.IndexSchema{
"id": &memdb.IndexSchema{
Name: "id",
Unique: true,
Indexer: &memdb.IntFieldIndex{Field: "Id"},
},
"name": &memdb.IndexSchema{
Name: "name",
Unique: false,
Indexer: &memdb.StringFieldIndex{Field: "Name"},
},
"score": &memdb.IndexSchema{
Name: "score",
Unique: false,
Indexer: &memdb.IntFieldIndex{Field: "Score"},
},
},
},
},
}
// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
panic(err)
}
// Create a write transaction
txn := db.Txn(true)
// Insert some people
people := []*Student{
&Student{1, "Anna", 95},
&Student{2, "Bob", 75},
&Student{3, "Dorothy", 80},
&Student{4, "Daniel", 90},
}
// insert to db
for _, p := range people {
if err := txn.Insert("student", p); err != nil {
panic(err)
}
}
// Commit the transaction
txn.Commit()
// Create read-only transaction
txn = db.Txn(false)
defer txn.Abort()
// query by name
raw, err := txn.First("student", "name", "Anna")
if err != nil {
panic(err)
}
// Say hi!
fmt.Printf("Hello student %s!\n", raw.(*Student).Name)
// query all the people
it, err := txn.Get("student", "name")
if err != nil {
panic(err)
}
fmt.Println("All the students:")
for obj := it.Next(); obj != nil; obj = it.Next() {
p := obj.(*Student)
fmt.Printf(" %s\n", p.Name)
}
// Range scan over students with score between 75 and 90 inclusive
it, err = txn.LowerBound("student", "score", 75)
if err != nil {
panic(err)
}
fmt.Println("Student with score 75 - 90:")
for obj := it.Next(); obj != nil; obj = it.Next() {
p := obj.(*Student)
if p.Score > 90 {
break
}
fmt.Printf(" %s with score %d\n", p.Name, p.Score)
}
}Output:
Hello student Anna!
All the students:
Anna
Bob
Daniel
Dorothy
Student with score 75 - 90:
Bob with score 75
Dorothy with score 80
Daniel with score 90
Deleting records in memdb
We can use Delete() function to remove 1 record, for multiple we will use deletion use DeleteAll() function.
func (txn *Txn) Delete(table string, obj interface{}) error: Delete is used to delete a single object from the given table. This object must already exist in the table
func (txn *Txn) DeleteAll(table, index string, args ...interface{}) (int, error): DeleteAll is used to delete all the objects in a given table matching the constraints on the index
Here is an example of deleting the record with Id=1, then delete all the students with 90 score.
package main
import (
"fmt"
"github.com/hashicorp/go-memdb"
)
// Create a struct
type Student struct {
Id int
Name string
Score int
}
func main() {
// Create the DB schema
schema := &memdb.DBSchema{
Tables: map[string]*memdb.TableSchema{
"student": &memdb.TableSchema{
Name: "student",
Indexes: map[string]*memdb.IndexSchema{
"id": &memdb.IndexSchema{
Name: "id",
Unique: true,
Indexer: &memdb.IntFieldIndex{Field: "Id"},
},
"name": &memdb.IndexSchema{
Name: "name",
Unique: true,
Indexer: &memdb.StringFieldIndex{Field: "Name"},
},
"score": &memdb.IndexSchema{
Name: "score",
Unique: false,
Indexer: &memdb.IntFieldIndex{Field: "Score"},
},
},
},
},
}
// Create a new data base
db, err := memdb.NewMemDB(schema)
if err != nil {
panic(err)
}
// Create a write transaction
txn := db.Txn(true)
// Insert some students
people := []*Student{
&Student{1, "Anna", 80},
&Student{2, "Bob", 85},
&Student{3, "Clair", 90},
&Student{4, "Dorothy", 95},
&Student{5, "Caitlyn", 90},
}
for _, p := range people {
if err := txn.Insert("student", p); err != nil {
panic(err)
}
}
// Commit the transaction
txn.Commit()
printAll(db.Txn(false))
fmt.Println("deleting one")
txn = db.Txn(true)
err = txn.Delete("student", Student{Id: 1})
if err != nil {
panic(err)
}
txn.Commit()
printAll(db.Txn(false))
fmt.Println("deleting many")
txn = db.Txn(true)
_, err = txn.DeleteAll("student", "score", 90)
if err != nil {
panic(err)
}
txn.Commit()
printAll(db.Txn(false))
}
func printAll(txn *memdb.Txn) {
// List all the students
it, err := txn.Get("student", "id")
if err != nil {
panic(err)
}
fmt.Println("All the students:")
for obj := it.Next(); obj != nil; obj = it.Next() {
p := obj.(*Student)
fmt.Printf(" %s\n", p.Name)
}
}Output:
All the students:
Anna
Bob
Clair
Dorothy
Caitlyn
deleting one
All the students:
Bob
Clair
Dorothy
Caitlyn
deleting many
All the students:
Bob
Dorothy
Summary
In this article I have given some examples of working with memdb in golang. You can read more memdb functions in official documentation.
The database provides the following:
- Multi-Version Concurrency Control (MVCC) - By leveraging immutable radix trees the database is able to support any number of concurrent readers without locking, and allows a writer to make progress.
- Transaction Support - The database allows for rich transactions, in which multiple objects are inserted, updated or deleted. The transactions can span multiple tables, and are applied atomically. The database provides atomicity and isolation in ACID terminology, such that until commit the updates are not visible.
- Rich Indexing - Tables can support any number of indexes, which can be simple like a single field index, or more advanced compound field indexes. Certain types like UUID can be efficiently compressed from strings into byte indexes for reduced storage requirements.
- Watches - Callers can populate a watch set as part of a query, which can be used to detect when a modification has been made to the database which affects the query results. This lets callers easily watch for changes in the database in a very general way.
References
https://en.wikipedia.org/wiki/In-memory_database
https://github.com/hashicorp/go-memdb
