DRBD Tutorial | Setup Linux Disk Replication | CentOS 8


Storage

In this article I will share Step-by-Step DRBD 9.0 Installation and Configuration Guide on CentOS 8 using KVM based Virtual Machines. Below topics will be covered in this DRBD Tutorial

  • DRBD Installation
  • Configure global DRBD configuration
  • Create DRBD Resource and Device
  • Configure Linux Disk Replication
  • Manage DRBD primary and secondary nodes

 

Overview on DRBD

From LINBIT's DRBD 9.0 User Guide

DRBD (Distributed Replicated Block Device) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts.

DRBD mirrors data:

  • in real time: Replication occurs continuously while applications modify the data on the device.
  • transparently: Applications need not be aware that the data is stored on multiple hosts.
  • synchronously or asynchronously: With synchronous mirroring, applications are notified of write completions after the writes have been carried out on all (connected) hosts. With asynchronous mirroring, applications are notified of write completions when the writes have completed locally, which usually is before they have propagated to the other hosts

 

DRBD support for Red Hat and CentOS

 

Lab Environment

Below is my Lab Environment

  KVM Host KVM VM1 KVM VM2 KVM VM3
Hostname rhel-8 centos8-2 centos8-3 centos8-4
FQDN rhel-.example.com centos8-2.example.com centos8-3.example.com centos8-4.example.com
NIC 1 (Public IP) 10.43.138.12 NA NA NA
NIC 2 (Private IP) 192.168.122.1 (NAT) 192.168.122.10 192.168.122.11 192.168.122.12
OS RHEL 8.1 CentOS 8.1 CentOS 8.1 CentOS 8.1
Memory(GB) 128 10 10 10
Storage(GB) 500 40 40 40
Role PXE Server HA Cluster Node HA Cluster Node HA Cluster Node

 

Configure Lower Level Device for DRBD

To setup Linux Disk Replication using DRBD configuration in my DRBD Tutorial, I have added a new disk /dev/sdb of 5GB to all of my Cluster Nodes

[root@centos8-2 ~]# lsblk
NAME                   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda                      8:0    0  20G  0 disk
├─sda1                   8:1    0   1G  0 part /boot
└─sda2                   8:2    0  19G  0 part
  ├─cl_centos8--2-root 253:0    0  17G  0 lvm  /
  └─cl_centos8--2-swap 253:1    0   2G  0 lvm  [SWAP]
sdb                      8:16   0   5G  0 disk

 

This will become the lower-level device for your DRBD resource. You may use any type of block device found on your system for this purpose. Typical examples include:

  • A hard drive partition (or a full physical hard drive),
  • A software RAID device,
  • An LVM Logical Volume or any other block device configured by the Linux device-mapper infrastructure,
  • Any other block device type found on your system.

In our DRBD tutorial we will go ahead with Logical Volume solution for DRBD configuration

Create a new Physical Volume using /dev/sdb

[root@centos8-2 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

Currently there is no free space available in my Volume group

[root@centos8-2 ~]# vgs
  VG           #PV #LV #SN Attr   VSize   VFree
  cl_centos8-2   1   2   0 wz--n- <19.00g    0

I already have a Volume Group, so I will extend the Volume Group by adding the new Physical Volume on all the three Cluster nodes

[root@centos8-2 ~]# vgextend cl_centos8-2 /dev/sdb
  Volume group "cl_centos8-2" successfully extended

Next I will create a logical volume for my DRBD configuration to setup Linux disk replication across Cluster nodes

[root@centos8-2 ~]# lvcreate -L 4G -n drdb-1 cl_centos8-2
  Logical volume "drdb-1" created.
NOTE:
It is not mandatory to use same Volume Group and Logical Volume name for DRBD configuration. For example in my case, below are the logical volume path and names from 3 cluster nodes which are all under different VG, although LV name is same but again that is also not mandatory.
/dev/cl_centos8-2/drdb-1
/dev/cl_centos8-3/drdb-1
/dev/cl_centos8-4/drdb-1

 

Install DRBD 9.0 from ELREPO

  • By default ELREPO will not be available on CentOS server.
  • To install ELREPO, first you should import the GPG key or else the GPG check will fail later.
  • Since my Cluster nodes are in private network with no internet access, I have created an offline repository using the elrepo packages.
  • This repository is shared over my private network using NFS server
  • You can directly go ahead and install elpo-release rpm on all you servers where you plan to setup Linux disk replication
[root@repo-server ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org

Next install the ELPO repo for CentOS 8 release

[root@repo-server ~]# rpm -Uvh  https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
Retrieving https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm
Verifying...                          ################################# [100%]
Preparing...                          ################################# [100%]
Updating / installing...
   1:elrepo-release-8.1-1.el8.elrepo  ################################# [100%]

For RHEL environment, you can create a repo file /etc/yum.repos.d/linbit.repo to reflect the following changes.

[drbd-9.0]
name=DRBD 9.0
baseurl=http://packages.linbit.com/<hash>/yum/rhel7/drbd-9.0/<arch>
gpgcheck=0
IMPORTANT NOTE:
You will have to populate the <hash> and <arch> variables. This is provided by LINBIT support services.

Once the elpo-release rpm is installed on all your Cluster nodes, use dnf to lookup drbd package

[root@centos8-2 ~]# dnf search drbd
============================================================== Name & Summary Matched: drbd ===============================================================
kmod-drbd90.x86_64 : drbd90 kernel module(s)
collectd-drbd.x86_64 : DRBD plugin for collectd
drbd90-utils.x86_64 : Management utilities for DRBD
drbd90-utils-sysvinit.x86_64 : The SysV initscript to manage the DRBD.
drbdlinks.noarch : Program for managing links into a DRBD shared partition

So there are the available drbd package.

We will install kmod-drbd90 and drbd90-utils on all the Cluster nodes

[root@centos8-2 ~]# dnf install kmod-drbd90.x86_64 drbd90-utils.x86_64 -y

 

DRBD configuration

All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf. Normally, this configuration file is just a skeleton with the following contents:

[root@centos8-2 ~]# cat /etc/drbd.conf
# You can find an example in  /usr/share/doc/drbd.../drbd.conf.example

include "drbd.d/global_common.conf";
include "drbd.d/*.res";

By convention, /etc/drbd.d/global_common.conf contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.

 

Sample DRBD Configuration File

You will get a sample DRBD configuration file under /usr/share/doc/packages/drbd. I will attach a copy of this drbd configuration file for reference

drbd.conf.example

 

Configure global_common.conf

We will have a simple global_common.conf with below entries in our DRBD tutorial:

[root@centos8-2 ~]# cat /etc/drbd.d/global_common.conf
global {
 usage-count no;
}
common {
 net {
  protocol C;
 }
}

 

usage-count

  • The usage-count no line in the global section skips sending a notice to the DRBD team each time a new version of the software is installed in your system.
  • You could change it to yes if you want to submit information from your system.
  • Alternatively, you could change it to ask if you want to be prompted for a decision each time you do an upgrade.
  • Either way, you should know that they use this information for statistical analysis only, and their reports are always available to the public at http://usage.drbd.org/cgi-bin/show_usage.pl.

protocol C

  • The protocol C line tells the DRBD resource to use a fully synchronous replication
  • This means that local write operations on the node that is functioning as primary are considered completed only after both the local and remote disk writes have been confirmed.
  • Thus, if you run into the loss of a single node, that should not lead to any data loss under normal circumstances, unless both nodes (or their storage subsystems) are irreversibly destroyed at the same time.

Copy this global_common.conf file to all the cluster nodes

[root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-3:/etc/drbd.d/
[root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-4:/etc/drbd.d/

 

Create DRBD resource

  • A per-resource configuration file is usually named /etc/drbd.d/<resource>.res.
  • Any DRBD resource you define must be named by specifying a resource name in the configuration.
  • In our DRBD Tutorial we will name our resource as drbd1
  • Every resource configuration must also have at least two on host sub-sections, one for every cluster node.
  • All other configuration settings are either inherited from the common section (if it exists), or derived from DRBD’s default settings.

Below is my sample DRBD resource configuration file

[root@centos8-2 ~]# cat /etc/drbd.d/drbd1.res
resource drbd1 {
 meta-disk internal;
 device /dev/drbd1;
 net {
  verify-alg sha256;
 }
 on centos8-2.example.com {   		<-- hostname must match `uname -n` output
  node-id   0;				<-- Assign a unique node-id
  disk   /dev/cl_centos8-2/drdb-1;	<-- Logical Volume on the provided host
  address  192.168.122.10:7789;		<-- IP Address to be used to connect to the node with port
 }
 on centos8-3.example.com {
  node-id   1;
  disk   /dev/cl_centos8-3/drdb-1;
  address  192.168.122.11:7789;
 }
 on centos8-4.example.com {
  node-id   2;
  disk  /dev/cl_centos8-4/drdb-1;
  address  192.168.122.12:7789;
 }
 connection-mesh {
  hosts centos8-2 centos8-3 centos8-4;
 }
}
  • By default DRBD uses port between 7788-7790, we have explicitly defined port number as 7789 for internal communication in our DRBD Tutorial
  • We must make sure this port is reachable over firewall
  • We have a global section where we provide the values which are common across all the servers
  • Our DRBD device resource would be /dev/drbd1 i.e. the logical volumes from all the cluster nodes would use /dev/drbd1 to perform Linux disk replication
  • On a Linux server with DRBD, you can see the available devices under /dev/drbd*
  • We will use sha256 algorithm for sync verification. You can choose any digest algorithm supported by the kernel crypto API in your system’s kernel configuration

Follow drbd9.X man page to get the complete list of supported arguments. Here,

 on host-name [...]			
							Define the properties of a resource on a particular host or set of hosts.
							The host-name argument must match the Linux host name (uname -n).

 node-id value				
							Defines the unique node identifier for a node in the cluster. 
							The node-id parameter exists since DRBD 9. Its value ranges from 0 to 16; 
							there is nodefault.
							
 disk {[disk] | none}		
							Define the lower-level block device that DRBD will use for storing the actual data. 
							While the replicated drbd device is configured, the lower-level device must not be
							used directly. 
							
 address [address-family] address:port
							Defines the address family, address, and port of a connection endpoint.

 

Create and Enable DRBD Resource

Next we must copy the DRBD resource file to all the cluster nodes

[root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-3:/etc/drbd.d/
root@centos8-3's password:
drbd1.res                                                                                                                100%  492   291.8KB/s   00:00

[root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-4:/etc/drbd.d/
root@centos8-4's password:
drbd1.res                                                                                                                100%  492   376.7KB/s   00:00
  • After you have completed initial resource configuration as outlined in the previous sections, you can bring up your resource.
  • When we installed DRBD earlier, a utility called drbdadm was installed as well
  • drbdadm is intended to be used for the administration of DRBD resources, such as our newly configured volume
  • The first step of DRBD Tutorial and Linux Disk Replication in starting and bringing a DRBD resource online is to initialize its metadata
  • Note that the /var/lib/drbd directory is needed beforehand. If it was not created previously when you installed DRBD, create it manually before proceeding
# mkdir /var/lib/drbd

Each of the following steps must be completed on all the cluster nodes.

[root@centos8-2 ~]# drbdadm create-md drbd1

md_offset 4294963200
al_offset 4294930432
bm_offset 4294799360

Found some data

 ==> This might destroy existing data! <==

Do you want to proceed?
[need to type 'yes' to confirm] yes

initializing activity log
initializing bitmap (128 KB) to all zero
Writing meta data...
New drbd meta data block successfully created.

So our DRBD device /dev/drbd1 is ready. Repeat the same steps on other Cluster nodes for DRBD configuration

[root@centos8-3 ~]# drbdadm create-md drbd1
[root@centos8-4 ~]# drbdadm create-md drbd1

 

Setup Firewall for DRBD

Since we used custom port 7789 for communication between all the cluster nodes in the DRBD configuration, we must allow port 7789 in the firewall rule on all the cluster nodes to setup Linux disk replication across servers

[root@centos8-3 ~]# firewall-cmd --add-port 7789/tcp --permanent
[root@centos8-3 ~]# firewall-cmd --reload

 

Enable DRBD Device

The next step consists of drbd configuration includes enabling drbd1 resource in order to finish allocating both disk and network resources for its operation:

[root@centos8-2 ~]# drbdadm up drbd1
[root@centos8-3 ~]# drbdadm up drbd1
[root@centos8-4 ~]# drbdadm up drbd1

 

Check DRBD Device status

drbdadm status output should now contain information similar to the following related to Linux Disk Replication:

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Secondary
  disk:Inconsistent
  centos8-3.example.com connection:Inconsistent
  centos8-4.example.com connection:Inconsistent

By now, DRBD has successfully allocated both disk and network resources and is ready for operation in our DRBD Tutorial.

NOTE:
If you get "connecting" in the status for peer KVM nodes then it means, "This node is waiting until the peer node becomes visible on the network". Check your firewall rules to make sure your firewall is not blocking the connection. You can temporarily disable firewall by using "systemctl stop firewalld" on the impacted KVM cluster nodes and restart the DRBD resource using drbdadm down <resource> followed by drbdadm up <resource>. If the drbd state changes then it means the firewall is most likely your problem. Next you must check your firewall policy and system logs /var/log/messages for more debug information.

 

Configure DRDB Primary server

  • Note that the status of the device shows as "inconsistent" since we haven't indicated yet which of the DRBD devices will act as a primary device and which one as a secondary device.
  • This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source.
  • To perform this step, issue this command from the server which you want to act as primary node
[root@centos8-2 ~]# drbdadm primary --force drbd1
  • After issuing this command, the initial full synchronization will commence the Linux Disk Replication.
  • You will be able to monitor its progress via drbdadm status <resource>.
  • It may take some time for drbd synchronization depending on the size of the device.

So our DRBD device will consider centos8-2 as primary and will sync the data of /dev/drbd1 across other KVM nodes

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Primary
  disk:UpToDate
  centos8-3.example.com role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:2.91
  centos8-4.example.com role:Secondary
    replication:SyncSource peer-disk:Inconsistent done:2.88

By now, your DRBD device is fully operational, even before the Linux disk replication has completed. Checking the status after few minutes

[root@centos8-2 ~]# drbdadm status drbd1
drbd1 role:Primary
  disk:UpToDate
  centos8-3.example.com role:Secondary
    peer-disk:UpToDate
  centos8-4.example.com role:Secondary
    peer-disk:UpToDate

You can check the device mapping across all the Cluster nodes

[root@centos8-2 ~]# ls -l /dev/drbd/by-disk/cl_centos8-2/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:05 /dev/drbd/by-disk/cl_centos8-2/drdb-1 -> ../../../drbd1

[root@centos8-3 ~]# ls -l /dev/drbd/by-disk/cl_centos8-3/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:06 /dev/drbd/by-disk/cl_centos8-3/drdb-1 -> ../../../drbd1

[root@centos8-4 ~]# ls -l /dev/drbd/by-disk/cl_centos8-4/drdb-1
lrwxrwxrwx 1 root root 14 May  2 12:06 /dev/drbd/by-disk/cl_centos8-4/drdb-1 -> ../../../drbd1

 

Enabling Dual-primary Node (Optional)

  • Dual-primary mode allows a resource to assume the primary role simultaneously on more than one node.
  • Dual-primary mode requires that the resource is configured to replicate synchronously (protocol C). Because of this it is latency sensitive, and ill suited for WAN environments.
  • Additionally, as both resources are always primary, any interruption in the network between KVM nodes will result in a split-brain.
  • In this DRBD Tutorial, we will use single primary node and the steps are only for demonstration
  • To enable dual-primary mode, set the allow-two-primaries option to yes in the net section of your resource configuration:
resource <resource>
  net {
    protocol C;
    allow-two-primaries yes;
    fencing resource-and-stonith;
  }
  • Next update this configuration file on all the cluster nodes
  • If you make this change to an existing resource, as always, synchronize your drbd.conf to the peer,
  • Execute below command on all the cluster nodes
# drbdadm adjust <resource>

 

Create File System on DRBD device

  • Next we will create a file system on our DRBD device to be able to perform read write operations.
  • I will create ext4 file system on /dev/drbd1
  • This step is required on any one of the Cluster nodes and the changes will be replicated across as our DRBD is already running
[root@centos8-2 ~]# mkfs.ext4 /dev/drbd1
mke2fs 1.44.6 (5-Mar-2019)
Discarding device blocks: done
Creating filesystem with 1048503 4k blocks and 262144 inodes
Filesystem UUID: e4f134c7-6ef5-4508-9d11-358a030e19a8
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Allocating group tables: done
Writing inode tables: done
Creating journal (16384 blocks): done
Writing superblocks and filesystem accounting information: done

Now, the DRBD resource is ready to be used in our DRBD tutorial as usual. You can now mount it on a mount point and start saving files to it.

Create a mount point with same name on all the cluster nodes

[root@centos8-2 ~]# mkdir /share
[root@centos8-3 ~]# mkdir /share
[root@centos8-4 ~]# mkdir /share

Next you can manually mount the share to the mount point on one of the Cluster nodes

[root@centos8-2 ~]# mount /dev/drbd1 /share
[root@centos8-2 ~]# df -h /share/
Filesystem      Size  Used Avail Use% Mounted on
/dev/drbd1      3.9G   16M  3.7G   1% /share

 

 

Lastly I hope the steps from this DRBD tutorial to install configure DRBD and setup Linux disk replication on CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.

 

References:
Linbit: Introduction to DRBD

Deepak Prasad

Deepak Prasad

He is the founder of GoLinuxCloud and brings over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, AWS, Networking, and Security. With extensive experience, he excels in various domains, from development to DevOps, Networking, and Security, ensuring robust and efficient solutions for diverse projects. You can connect with him on his LinkedIn profile.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

15 thoughts on “DRBD Tutorial | Setup Linux Disk Replication | CentOS 8”

  1. Hi,
    is it possible to configure user and group quota for drbd disk which is on LVM , RHEL 9?

    After mounted the drbd , encountered “quotacheck: Cannot find filesystem to check or filesystem not mounted with quota option.” with /etc/fstab additional config below

    /dev/drbdvg/lv_data1 /drbd_mnt ext4 defaults,noauto,usrquota,grpquota 0 0

    below sample of the drbd resource config:

    resource test {
            on testserver1 {
                    device /dev/drbd0;
                    disk /dev/drbdvg/lv_data1;
                            meta-disk /dev/drbdvg/lv_drbd_metadata;
                            address x.x.x.x:7789;
            }
    	on testserver2 {
                    device /dev/drbd0;
                    disk /dev/drbdvg/lv_data1;
                            meta-disk /dev/drbdvg/lv_drbd_metadata;
                            address x.x.x.y:7789;
            }
    }
    Reply
    • can you check if quota is applied to the filesystem

      mount | grep drbd_mnt
      quotacheck -cug /drbd_mnt
      quotaon -vug /drbd_mnt
      Reply
      • Hi,
        greatly appreciate , below is the results

        [root@testserver1 bin]# mount | grep drbd
        /dev/mapper/drbdvg-lv_data1 on /drbd_mnt type ext4 (rw,relatime,seclabel)
        
        [root@testserver1 bin]# quotacheck -cug /drbd_mnt/
        quotacheck: Mountpoint (or device) /drbd_mnt not found or has no quota enabled.
        quotacheck: Cannot find filesystem to check or filesystem not mounted with quota option.
        
        [root@testserver1 bin]# quotaon -vug /drbd_mnt/
        quotaon: Mountpoint (or device) /drbd_mnt not found or has no quota enabled.
        Reply
        • It seems quota is not activated on your filesystem. Since you fstab already has the required entries so you can try

          mount -a

          and then verify again with

          mount | grep drbd

          If you still don’t see quota flags then you can try to execute:

          mount -o remount /drbd_mnt
          Reply
          • after did the following, get to no results for user / group quota

            [root@testserver1 bin]# mount -a
            [root@testserver1 bin]# mount | grep drbd
            [root@testserver1 bin]#
            [root@testserver1 bin]# mount -o remount /drbd_mnt/
            mount: /drbd_mnt: mount point not mounted or bad option.
            [root@testserver1 bin]# mount /dev/drbdvg/lv_data1 /drbd_mnt
            mount: /drbd_mnt: /dev/mapper/drbdvg-lv_data1 already mounted or mount point busy.
            [root@testserver1 bin]#

            noticed that only after I mount /dev/drbd0 to /drbd_mnt, then quota can be configured. Thus would like to seek your experience if DRBD on LVM is possible for quota configuration?

            root@testserver1 bin]# mount /dev/drbd0 /drbd_mnt/
            root@testserver1 bin]# mount | grep drbd_mnt
            /dev/drbd0 on /drbd_mnt type ext4 (rw,relatime,seclabel)
            [root@testserver1 bin]# mount -a
            [root@testserver1 bin]# mount -o remount /drbd_mnt/
            [root@testserver1 bin]# mount | grep drbd_mnt
            /dev/drbd0 on /drbd_mnt type ext4 (rw,relatime,seclabel,quota,usrquota,grpquota)
            [root@testserver1 bin]#
            [root@testserver1 bin]# quotacheck -cug /drbd_mnt/
            [root@testserver1 bin]# quotaon -vug /drbd_mnt/
            quotaon: Your kernel probably supports ext4 quota feature but you are using external quota files. Please switch your filesystem to use ext4 quota feature as external quota files on ext4 are deprecated.
            /dev/drbd0 [/drbd_mnt]: group quotas turned on
            /dev/drbd0 [/drbd_mnt]: user quotas turned on
            [root@testserver1 bin]#
            Reply
  2. Awesome guide! thank you

    I was able to bring up a two node cluster, and both shows to be updated

    # drbdadm status drbd1
    drbd1 role:Primary
      disk:UpToDate
      kvm-srv-2 role:Secondary
        peer-disk:UpToDate

    But the txt file i’ve put into the mounted folder, e.g. /share, was not replicated into the secondary node. Am I missing something?

    Reply
    • If you plan to use Physical Volume as your DRBD resource then you must set write_cache_state = 0 and also delete /etc/lvm/cache/.cache to clear any existing stale cache

      Reply

Leave a Comment