Golang FlashDB Tutorial- In Memory Database


GO

Author: Tuan Nguyen
Reviewer: Deepak Prasad

In this tutorial, we will build a simple in-memory key-value store example in Go with FlashDB. FlashDB is a simple in-memory key/value store written entirely in Go. It persists data on disk, is ACID compliant and employs locking for multiple readers and a single writer. It allows for redis-like operations, SET, SORTED SET, HASH, and STRING are examples of data structures.

 

FlashDB architecture

Golang FlashDB Tutorial- In Memory Database

FlashDB is made up of easily understandable composable libraries. The goal is to connect the learning gap for anyone who is learning how to build a simple ACID database. FlashDB supports a variety of Redis commands. Redis uses the following data structures to implement various types:

  • Set: Redis Sets are unsorted string collections. It is also possible to perform a variety of other operations on sets, such as determining whether a given element already exists, performing the intersection, union, or difference of multiple sets, and so on. This is also done with a simple HashMap data structure.
  • ZSet: Sorted sets are a data type that resembles a cross between a Set and a Hash. Sorted sets, like sets, are composed of unique, non-repeating string elements, so they are also sets in some ways. While elements within sets are not ordered, each element in a sorted set is assigned a floating point value known as the score (this is why the type is also similar to a hash, since every element is mapped to a value).
  • String: The Redis String value type is the most basic type of value that can be associated with a Redis key. Because Redis keys are strings, using the string type as a value also maps a string to another string.. A simple implementation of Adaptive Radix Tree (ART) in Go, that scans could be done easily.
  • Hash: While hashes are useful for representing objects, the number of fields you can store in a hash has no practical limit (other than available memory), so you can use hashes in a variety of ways within your application. This is done with a very simple HashMap data structure.
  • Append Only Log: A simple immutable append only log in Go. aol is a rewrite of wal to allow only appends to a log.

 

Install GO FlashDB

Run the below command to install FlashDB and its dependencies:

$ go get github.com/arriqaaq/flashdb
go: finding module for package github.com/arriqaaq/flashdb
go: downloading github.com/arriqaaq/flashdb v0.1.6
go: found github.com/arriqaaq/flashdb in github.com/arriqaaq/flashdb v0.1.6
go: downloading github.com/arriqaaq/aol v0.1.2
go: downloading github.com/arriqaaq/art v0.1.1
go: downloading github.com/arriqaaq/hash v0.1.2
go: downloading github.com/arriqaaq/set v0.1.2
go: downloading github.com/arriqaaq/zset v0.1.2
go: downloading github.com/stretchr/testify v1.7.1
go: downloading github.com/davecgh/go-spew v1.1.0
go: downloading gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c
go: downloading github.com/armon/go-radix v1.0.0
go: downloading github.com/arriqaaq/skiplist v0.1.6

 

Create and Open a database

The primary object in FlashDB is a DB. To open or create your database, use the flashdb.New() function:

package main

import (
	"log"

	"github.com/arriqaaq/flashdb"
)

func main() {
	config := &flashdb.Config{Path: "/path", EvictionInterval: 10}
	db, err := flashdb.New(config)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()

}

EvictionInterval (in seconds): NoSync disables fsync after writes. This is less durable and puts the log at risk of data loss when there's a server crash.

Verify the path after executing the code:

$ ls -l /path/
total 8
drwxr-x---  2 root root 4096 Oct  4 00:42 ./
drwxr-xr-x 21 root root 4096 Oct  4 00:42 ../
-rw-r-----  1 root root    0 Oct  4 00:42 00000000000000000001

 

To open a database that does not persist to disk, leave the path empty.

config := &flashdb.Config{Path:"", EvictionInterval: 10}
flashdb.New(config)

 

Performing Different Operations on FlashDB

When you need to make changes to your data, you use a read/write transaction. At any given time, only one read/write transaction can be active. As a result, make sure to close it as soon as you are finished with it.

All reads and writes must take place within a transaction. FlashDB can only have one write transaction open at a time, but many concurrent read transactions are possible. Each transaction keeps a consistent view of the database. In other words, once a transaction has begun, other transactions cannot change the data for that transaction.

When a transaction fails, it rolls back and undoes all changes to the database that occurred during that transaction. When a read/write transaction completes successfully, all changes are saved to disk.

 

Create a database and add key value pair based data

When you need to make changes to your data, you use a read/write transaction. At any given time, only one read/write transaction can be active. As a result, make sure to close it as soon as you are finished with it. To set /get key-value pair, open a read/write transaction and use Update() function.

Here is an example of set multiple key-value records with FlashDB:

package main

import (
	"fmt"
	"log"

	"github.com/arriqaaq/flashdb"
)

func main() {
	config := &flashdb.Config{Path: "", EvictionInterval: 10}
	db, err := flashdb.New(config)
	if err != nil {
		log.Fatal(err)
	}
	defer db.Close()
 
        // define a map to contains key-value pair
	studentMap := map[string]string{"Anna": "anna@gmail.com", "Bob": "bob@gmail.com", "Cloe": "cloe@gmail.com", "Daniel": "deniel@gmail.com"}

	err = db.Update(func(tx *flashdb.Tx) error {
		for key, value := range studentMap {
                        // set key-value record
			err := tx.Set(key, value)
			return err
		}
		return nil
	})
	if err != nil {
		log.Fatal(err)
	}
}

 

Read the content from FlashDB Database

We can query key-value pair by using View() function:

	db.View(func(tx *flashdb.Tx) error {
		val, _ := tx.Get("Anna")
		fmt.Printf("value is %s\n", val)
		return nil
	})

Output:

value is anna@gmail.com

 

Delete a record from FlashDB Database

If you want to delete a record in FlashDB, you can use Delete() function:

	db.Update(func(tx *flashdb.Tx) error {
		var err error
		for key, value := range studentMap {
			err = tx.Set(key, value)
			if err != nil {
				panic(err)
			}
		}
		err = tx.Delete("Anna")
		return err
	})

	db.View(func(tx *flashdb.Tx) error {
		val, err := tx.Get("Anna")
		val2, err := tx.Get("Bob")
		if err != nil {
			return err
		}
		fmt.Printf("After delete : value is %s\n", val)
		fmt.Printf("Another value is %s\n", val2)
		return nil
	})

Output:

After delete : value is 
Another value is bob@gmail.com

 

Summary

In this article, I have given some examples of performing operations in flashdb. The functions of different data types are listed below:

String Hash Set ZSet
SET HSET SADD ZADD
GET HGET SISMEMBER ZSCORE
DELETE HGETALL SRANDMEMBER ZCARD
EXPIRE HDEL SREM ZRANK
TTL HEXISTS SMOVE ZREVRANK
HLEN SCARD ZRANGE
HKEYS SMEMBERS ZREVRANGE
HVALS SUNION ZREM
HCLEAR SDIFF ZGETBYRANK
SCLEAR ZREVGETBYRANK
ZSCORERANGE
ZREVSCORERANGE
ZCLEAR

We have a topic about memdb (a in-memory database) in Golang. Depending on the application, you may want to consider using the databases listed above.

 

References

https://en.wikipedia.org/wiki/In-memory_database
https://github.com/arriqaaq/flashdb

 

Tuan Nguyen

Tuan Nguyen

He is proficient in Golang, Python, Java, MongoDB, Selenium, Spring Boot, Kubernetes, Scrapy, API development, Docker, Data Scraping, PrimeFaces, Linux, Data Structures, and Data Mining. With expertise spanning these technologies, he develops robust solutions and implements efficient data processing and management strategies across various projects and platforms. You can connect with him on his LinkedIn profile.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment