In this article I will share Step-by-Step DRBD 9.0 Installation and Configuration Guide on CentOS 8 using KVM based Virtual Machines. Below topics will be covered in this DRBD Tutorial
- DRBD Installation
- Configure global DRBD configuration
- Create DRBD Resource and Device
- Configure Linux Disk Replication
- Manage DRBD primary and secondary nodes
Overview on DRBD
From LINBIT's DRBD 9.0 User Guide
DRBD (Distributed Replicated Block Device) is a software-based, shared-nothing, replicated storage solution mirroring the content of block devices (hard disks, partitions, logical volumes etc.) between hosts.
DRBD mirrors data:
- in real time:Â Replication occurs continuously while applications modify the data on the device.
- transparently: Applications need not be aware that the data is stored on multiple hosts.
- synchronously or asynchronously:Â With synchronous mirroring, applications are notified of write completions after the writes have been carried out on all (connected) hosts. With asynchronous mirroring, applications are notified of write completions when the writes have completed locally, which usually is before they have propagated to the other hosts
DRBD support for Red Hat and CentOS
- DRBD is provided officially by Linbit. You can get a free version of DRBD software but to get support, you must have a valid subscription with Linbit
- Red Hat does not provide DRBD package with their Red Hat Network but they do officially support DRBD if you have a valid subscription from Linbit
- For CentOS environment you can use ELPO repo to get the DRBD packages which is an open source community
- For this DRBD Tutorial we will use CentOS 8 with DRBD 9.0 to setup Linux Disk Replication
- Later we will use use this DRBD configuration with our pacemaker 2.0 cluster
Lab Environment
- I am using the same Lab Environment from my previous article where I configured my KVM HA Cluster using Pacemaker 2.0 on RHEL and CentOS 8
- I will not include the steps to install KVM and setup KVM High Availability Cluster in this DRBD tutorial as that would just make it very long
- So I will assume that you have a working High Availability Cluster to install and configure DRBD
- If you only intend to setup Linux Disk Replication using DRBD 9.0 then you do not need a Cluster
Below is my Lab Environment
KVM Host | KVM VM1 | KVM VM2 | KVM VM3 | |
---|---|---|---|---|
Hostname | rhel-8 | centos8-2 | centos8-3 | centos8-4 |
FQDN | rhel-.example.com | centos8-2.example.com | centos8-3.example.com | centos8-4.example.com |
NIC 1 (Public IP) | 10.43.138.12 | NA | NA | NA |
NIC 2 (Private IP) | 192.168.122.1 (NAT) | 192.168.122.10 | 192.168.122.11 | 192.168.122.12 |
OS | RHEL 8.1 | CentOS 8.1 | CentOS 8.1 | CentOS 8.1 |
Memory(GB) | 128 | 10 | 10 | 10 |
Storage(GB) | 500 | 40 | 40 | 40 |
Role | PXE Server | HA Cluster Node | HA Cluster Node | HA Cluster Node |
Configure Lower Level Device for DRBD
To setup Linux Disk Replication using DRBD configuration in my DRBD Tutorial, I have added a new disk /dev/sdb
of 5GB to all of my Cluster Nodes
[root@centos8-2 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT sda 8:0 0 20G 0 disk ├─sda1 8:1 0 1G 0 part /boot └─sda2 8:2 0 19G 0 part ├─cl_centos8--2-root 253:0 0 17G 0 lvm / └─cl_centos8--2-swap 253:1 0 2G 0 lvm [SWAP] sdb 8:16 0 5G 0 disk
This will become the lower-level device for your DRBD resource. You may use any type of block device found on your system for this purpose. Typical examples include:
- A hard drive partition (or a full physical hard drive),
- A software RAID device,
- An LVM Logical Volume or any other block device configured by the Linux device-mapper infrastructure,
- Any other block device type found on your system.
In our DRBD tutorial we will go ahead with Logical Volume solution for DRBD configuration
Create a new Physical Volume using /dev/sdb
[root@centos8-2 ~]# pvcreate /dev/sdb
Physical volume "/dev/sdb" successfully created.
Currently there is no free space available in my Volume group
[root@centos8-2 ~]# vgs VG #PV #LV #SN Attr VSize VFree cl_centos8-2 1 2 0 wz--n- <19.00g 0
I already have a Volume Group, so I will extend the Volume Group by adding the new Physical Volume on all the three Cluster nodes
[root@centos8-2 ~]# vgextend cl_centos8-2 /dev/sdb
Volume group "cl_centos8-2" successfully extended
Next I will create a logical volume for my DRBD configuration to setup Linux disk replication across Cluster nodes
[root@centos8-2 ~]# lvcreate -L 4G -n drdb-1 cl_centos8-2
Logical volume "drdb-1" created.
/dev/cl_centos8-2/drdb-1 /dev/cl_centos8-3/drdb-1 /dev/cl_centos8-4/drdb-1
Install DRBD 9.0 from ELREPO
- By default ELREPO will not be available on CentOS server.
- To install ELREPO, first you should import the GPG key or else the GPG check will fail later.
- Since my Cluster nodes are in private network with no internet access, I have created an offline repository using the elrepo packages.
- This repository is shared over my private network using NFS server
- You can directly go ahead and install elpo-release rpm on all you servers where you plan to setup Linux disk replication
[root@repo-server ~]# rpm --import https://www.elrepo.org/RPM-GPG-KEY-elrepo.org
Next install the ELPO repo for CentOS 8 release
Linbit
[root@repo-server ~]# rpm -Uvh https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm Retrieving https://www.elrepo.org/elrepo-release-8.el8.elrepo.noarch.rpm Verifying... ################################# [100%] Preparing... ################################# [100%] Updating / installing... 1:elrepo-release-8.1-1.el8.elrepo ################################# [100%]
For RHEL environment, you can create a repo file /etc/yum.repos.d/linbit.repo
to reflect the following changes.
[drbd-9.0] name=DRBD 9.0 baseurl=http://packages.linbit.com/<hash>/yum/rhel7/drbd-9.0/<arch> gpgcheck=0
<hash>
and <arch>
variables. This is provided by LINBIT support services.Once the elpo-release
rpm is installed on all your Cluster nodes, use dnf
to lookup drbd
package
[root@centos8-2 ~]# dnf search drbd ============================================================== Name & Summary Matched: drbd =============================================================== kmod-drbd90.x86_64 : drbd90 kernel module(s) collectd-drbd.x86_64 : DRBD plugin for collectd drbd90-utils.x86_64 : Management utilities for DRBD drbd90-utils-sysvinit.x86_64 : The SysV initscript to manage the DRBD. drbdlinks.noarch : Program for managing links into a DRBD shared partition
So there are the available drbd package.
We will install kmod-drbd90
and drbd90-utils
on all the Cluster nodes
[root@centos8-2 ~]# dnf install kmod-drbd90.x86_64 drbd90-utils.x86_64 -y
DRBD configuration
All aspects of DRBD are controlled in its configuration file, /etc/drbd.conf
. Normally, this configuration file is just a skeleton with the following contents:
[root@centos8-2 ~]# cat /etc/drbd.conf # You can find an example in /usr/share/doc/drbd.../drbd.conf.example include "drbd.d/global_common.conf"; include "drbd.d/*.res";
By convention, /etc/drbd.d/global_common.conf
contains the global and common sections of the DRBD configuration, whereas the .res files contain one resource section each.
Sample DRBD Configuration File
You will get a sample DRBD configuration file under /usr/share/doc/packages/drbd
. I will attach a copy of this drbd configuration file for reference
Configure global_common.conf
We will have a simple global_common.conf
with below entries in our DRBD tutorial:
[root@centos8-2 ~]# cat /etc/drbd.d/global_common.conf global { usage-count no; } common { net { protocol C; } }
usage-count
- The usage-count no line in the global section skips sending a notice to the DRBD team each time a new version of the software is installed in your system.
- You could change it to yes if you want to submit information from your system.
- Alternatively, you could change it to ask if you want to be prompted for a decision each time you do an upgrade.
- Either way, you should know that they use this information for statistical analysis only, and their reports are always available to the public at http://usage.drbd.org/cgi-bin/show_usage.pl.
protocol C
- The protocol C line tells the DRBD resource to use a fully synchronous replication
- This means that local write operations on the node that is functioning as primary are considered completed only after both the local and remote disk writes have been confirmed.
- Thus, if you run into the loss of a single node, that should not lead to any data loss under normal circumstances, unless both nodes (or their storage subsystems) are irreversibly destroyed at the same time.
Copy this global_common.conf
file to all the cluster nodes
[root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-3:/etc/drbd.d/ [root@centos8-2 ~]# scp /etc/drbd.d/global_common.conf centos8-4:/etc/drbd.d/
Create DRBD resource
- A per-resource configuration file is usually named
/etc/drbd.d/<resource>.res
. - Any DRBD resource you define must be named by specifying a resource name in the configuration.
- In our DRBD Tutorial we will name our resource as
drbd1
- Every resource configuration must also have at least two on host sub-sections, one for every cluster node.
- All other configuration settings are either inherited from the common section (if it exists), or derived from DRBD’s default settings.
Below is my sample DRBD resource configuration file
[root@centos8-2 ~]# cat /etc/drbd.d/drbd1.res resource drbd1 { meta-disk internal; device /dev/drbd1; net { verify-alg sha256; } on centos8-2.example.com { <-- hostname must match `uname -n` output node-id 0; <-- Assign a unique node-id disk /dev/cl_centos8-2/drdb-1; <-- Logical Volume on the provided host address 192.168.122.10:7789; <-- IP Address to be used to connect to the node with port } on centos8-3.example.com { node-id 1; disk /dev/cl_centos8-3/drdb-1; address 192.168.122.11:7789; } on centos8-4.example.com { node-id 2; disk /dev/cl_centos8-4/drdb-1; address 192.168.122.12:7789; } connection-mesh { hosts centos8-2 centos8-3 centos8-4; } }
- By default DRBD uses port between 7788-7790, we have explicitly defined port number as 7789 for internal communication in our DRBD Tutorial
- We must make sure this port is reachable over firewall
- We have a global section where we provide the values which are common across all the servers
- Our DRBD device resource would be
/dev/drbd1
i.e. the logical volumes from all the cluster nodes would use/dev/drbd1
to perform Linux disk replication - On a Linux server with DRBD, you can see the available devices under
/dev/drbd*
- We will use sha256 algorithm for sync verification. You can choose any digest algorithm supported by the kernel crypto API in your system’s kernel configuration
Follow drbd9.X man page to get the complete list of supported arguments. Here,
on host-name [...] Define the properties of a resource on a particular host or set of hosts. The host-name argument must match the Linux host name (uname -n). node-id value Defines the unique node identifier for a node in the cluster. The node-id parameter exists since DRBD 9. Its value ranges from 0 to 16; there is nodefault. disk {[disk] | none} Define the lower-level block device that DRBD will use for storing the actual data. While the replicated drbd device is configured, the lower-level device must not be used directly. address [address-family] address:port Defines the address family, address, and port of a connection endpoint.
Create and Enable DRBD Resource
Next we must copy the DRBD resource file to all the cluster nodes
[root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-3:/etc/drbd.d/ root@centos8-3's password: drbd1.res 100% 492 291.8KB/s 00:00 [root@centos8-2 ~]# scp /etc/drbd.d/drbd1.res centos8-4:/etc/drbd.d/ root@centos8-4's password: drbd1.res 100% 492 376.7KB/s 00:00
- After you have completed initial resource configuration as outlined in the previous sections, you can bring up your resource.
- When we installed DRBD earlier, a utility called
drbdadm
was installed as well drbdadm
is intended to be used for the administration of DRBD resources, such as our newly configured volume- The first step of DRBD Tutorial and Linux Disk Replication in starting and bringing a DRBD resource online is to initialize its metadata
- Note that the
/var/lib/drbd
directory is needed beforehand. If it was not created previously when you installed DRBD, create it manually before proceeding
# mkdir /var/lib/drbd
Each of the following steps must be completed on all the cluster nodes.
[root@centos8-2 ~]# drbdadm create-md drbd1 md_offset 4294963200 al_offset 4294930432 bm_offset 4294799360 Found some data ==> This might destroy existing data! <== Do you want to proceed? [need to type 'yes' to confirm] yes initializing activity log initializing bitmap (128 KB) to all zero Writing meta data... New drbd meta data block successfully created.
So our DRBD device /dev/drbd1
is ready. Repeat the same steps on other Cluster nodes for DRBD configuration
[root@centos8-3 ~]# drbdadm create-md drbd1 [root@centos8-4 ~]# drbdadm create-md drbd1
connection
or connection-mesh
then you may get "Use explicit 'connection' sections (or a 'connection-mesh' section) with more than two 'on' sections.
" for your resource while setting up drbd 9.X
Setup Firewall for DRBD
Since we used custom port 7789 for communication between all the cluster nodes in the DRBD configuration, we must allow port 7789 in the firewall rule on all the cluster nodes to setup Linux disk replication across servers
[root@centos8-3 ~]# firewall-cmd --add-port 7789/tcp --permanent [root@centos8-3 ~]# firewall-cmd --reload
Enable DRBD Device
The next step consists of drbd configuration includes enabling drbd1
resource in order to finish allocating both disk and network resources for its operation:
[root@centos8-2 ~]# drbdadm up drbd1 [root@centos8-3 ~]# drbdadm up drbd1 [root@centos8-4 ~]# drbdadm up drbd1
Check DRBD Device status
drbdadm
status output should now contain information similar to the following related to Linux Disk Replication:
[root@centos8-2 ~]# drbdadm status drbd1 drbd1 role:Secondary disk:Inconsistent centos8-3.example.com connection:Inconsistent centos8-4.example.com connection:Inconsistent
By now, DRBD has successfully allocated both disk and network resources and is ready for operation in our DRBD Tutorial.
connecting
" in the status for peer KVM nodes then it means, "This node is waiting until the peer node becomes visible on the network". Check your firewall rules to make sure your firewall is not blocking the connection. You can temporarily disable firewall by using "systemctl stop firewalld
" on the impacted KVM cluster nodes and restart the DRBD resource using drbdadm down <resource>
followed by drbdadm up <resource>
. If the drbd state changes then it means the firewall is most likely your problem. Next you must check your firewall policy and system logs /var/log/messages
for more debug information.
Configure DRDB Primary server
- Note that the status of the device shows as "
inconsistent
" since we haven't indicated yet which of the DRBD devices will act as a primary device and which one as a secondary device. - This step must be performed on only one node, only on initial resource configuration, and only on the node you selected as the synchronization source.
- To perform this step, issue this command from the server which you want to act as primary node
[root@centos8-2 ~]# drbdadm primary --force drbd1
- After issuing this command, the initial full synchronization will commence the Linux Disk Replication.
- You will be able to monitor its progress via
drbdadm status <resource>
. - It may take some time for
drbd
synchronization depending on the size of the device.
So our DRBD device will consider centos8-2
as primary and will sync the data of /dev/drbd1
across other KVM nodes
[root@centos8-2 ~]# drbdadm status drbd1 drbd1 role:Primary disk:UpToDate centos8-3.example.com role:Secondary replication:SyncSource peer-disk:Inconsistent done:2.91 centos8-4.example.com role:Secondary replication:SyncSource peer-disk:Inconsistent done:2.88
By now, your DRBD device is fully operational, even before the Linux disk replication has completed. Checking the status after few minutes
[root@centos8-2 ~]# drbdadm status drbd1 drbd1 role:Primary disk:UpToDate centos8-3.example.com role:Secondary peer-disk:UpToDate centos8-4.example.com role:Secondary peer-disk:UpToDate
You can check the device mapping across all the Cluster nodes
[root@centos8-2 ~]# ls -l /dev/drbd/by-disk/cl_centos8-2/drdb-1 lrwxrwxrwx 1 root root 14 May 2 12:05 /dev/drbd/by-disk/cl_centos8-2/drdb-1 -> ../../../drbd1 [root@centos8-3 ~]# ls -l /dev/drbd/by-disk/cl_centos8-3/drdb-1 lrwxrwxrwx 1 root root 14 May 2 12:06 /dev/drbd/by-disk/cl_centos8-3/drdb-1 -> ../../../drbd1 [root@centos8-4 ~]# ls -l /dev/drbd/by-disk/cl_centos8-4/drdb-1 lrwxrwxrwx 1 root root 14 May 2 12:06 /dev/drbd/by-disk/cl_centos8-4/drdb-1 -> ../../../drbd1
Enabling Dual-primary Node (Optional)
- Dual-primary mode allows a resource to assume the primary role simultaneously on more than one node.
- Dual-primary mode requires that the resource is configured to replicate synchronously (protocol C). Because of this it is latency sensitive, and ill suited for WAN environments.
- Additionally, as both resources are always primary, any interruption in the network between KVM nodes will result in a split-brain.
- In this DRBD Tutorial, we will use single primary node and the steps are only for demonstration
- To enable dual-primary mode, set the
allow-two-primaries
option toyes
in the net section of your resource configuration:
resource <resource> net { protocol C; allow-two-primaries yes; fencing resource-and-stonith; }
- Next update this configuration file on all the cluster nodes
- If you make this change to an existing resource, as always, synchronize your
drbd.conf
to the peer, - Execute below command on all the cluster nodes
# drbdadm adjust <resource>
Create File System on DRBD device
- Next we will create a file system on our DRBD device to be able to perform read write operations.
- I will create
ext4
file system on/dev/drbd1
- This step is required on any one of the Cluster nodes and the changes will be replicated across as our DRBD is already running
[root@centos8-2 ~]# mkfs.ext4 /dev/drbd1 mke2fs 1.44.6 (5-Mar-2019) Discarding device blocks: done Creating filesystem with 1048503 4k blocks and 262144 inodes Filesystem UUID: e4f134c7-6ef5-4508-9d11-358a030e19a8 Superblock backups stored on blocks: 32768, 98304, 163840, 229376, 294912, 819200, 884736 Allocating group tables: done Writing inode tables: done Creating journal (16384 blocks): done Writing superblocks and filesystem accounting information: done
Now, the DRBD resource is ready to be used in our DRBD tutorial as usual. You can now mount it on a mount point and start saving files to it.
Create a mount point with same name on all the cluster nodes
[root@centos8-2 ~]# mkdir /share [root@centos8-3 ~]# mkdir /share [root@centos8-4 ~]# mkdir /share
Next you can manually mount the share to the mount point on one of the Cluster nodes
[root@centos8-2 ~]# mount /dev/drbd1 /share
[root@centos8-2 ~]# df -h /share/
Filesystem Size Used Avail Use% Mounted on
/dev/drbd1 3.9G 16M 3.7G 1% /share
Lastly I hope the steps from this DRBD tutorial to install configure DRBD and setup Linux disk replication on CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.
References:
Linbit: Introduction to DRBD
Hi,
is it possible to configure user and group quota for drbd disk which is on LVM , RHEL 9?
After mounted the drbd , encountered “
quotacheck: Cannot find filesystem to check or filesystem not mounted with quota option.
” with/etc/fstab
additional config belowbelow sample of the drbd resource config:
can you check if quota is applied to the filesystem
Hi,
greatly appreciate , below is the results
It seems quota is not activated on your filesystem. Since you fstab already has the required entries so you can try
and then verify again with
If you still don’t see quota flags then you can try to execute:
after did the following, get to no results for user / group quota
noticed that only after I mount
/dev/drbd0
to/drbd_mnt
, then quota can be configured. Thus would like to seek your experience if DRBD on LVM is possible for quota configuration?I am afraid I never tested with quota on DRBD so can’t give you a first hand feedback.
noted , and greatly appreciate your prompt response in the queries!
Awesome guide! thank you
I was able to bring up a two node cluster, and both shows to be updated
But the txt file i’ve put into the mounted folder, e.g. /share, was not replicated into the secondary node. Am I missing something?
This is weird, did you checked the logs for any information.
Thanks man, I solved by removing DRBD resource and restarting from scratch from “Create and Enable DRBD Resource” step.
Then the two nodes did successfully sync, and I was able to see the replicated .txt after switching secondary node to be primary in order to allow to mount folder on second host.
I am glad it worked.
Hi…
do we need to disable lvmwrite_cache and add drbd device filters in lvm.conf
If you plan to use Physical Volume as your DRBD resource then you must set
write_cache_state = 0
and also delete/etc/lvm/cache/.cache
to clear any existing stale cacheI have one doubt Can we do drbd both Centos and Ubuntu…?
Do you mean setup a DRBD cluster with one node on CentOS and other on Ubuntu? Please share all relevant details.