How to configure High Availability LVM i.e. HA LVM on CentOS or RHEL 7 Linux without GFS2. How can I configure my cluster service or resource to activate and monitor shared LVM volumes? Uncontrolled simultaneous access to shared storage can lead to data corruption. Storage access must be managed like any other active/passive service - it must only be active on a single machine at a time. Is there a way to configure LVM volumes for active/passive use in a High Availability cluster?
To configure HA LVM, as the name suggests you need logical volumes. So make sure you have logical volumes and volume groups with the same name on all your nodes of the cluster.
Earlier I had shared an article about Cluster Architecture and Types of Clusters and also step by step guide to configure HA Cluster with three nodes. Now later I had to remove one of the cluster node to demonstrate you about two node cluster setup and it's configuration.
So I will continue to use the same setup for demonstration of this article. As you see I already have some resource groups which are part of my cluster. I have written another article to help you understand all about resource groups and resource constraints on Cluster.
[root@node1 ~]# pcs status Cluster name: mycluster Stack: corosync Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Wed Dec 26 18:37:51 2018 Last change: Wed Dec 26 18:37:29 2018 by root via cibadmin on node1.example.com 2 nodes configured 6 resources configured Online: [ node1.example.com node2.example.com ] Full list of resources: Resource Group: apache-group apache-ip (ocf::heartbeat:IPaddr2): Started node2.example.com apache-service (ocf::heartbeat:apache): Started node2.example.com Resource Group: ftp-group ftp-ip (ocf::heartbeat:IPaddr2): Stopped ftp-service (systemd:vsftpd): Stopped Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
Setting up the environment
My cluster is running on RHEL 7 installed on Oracle VirtualBox. You can install Oracle VirtualBox also on your RHEL/CentOS Linux host.
Now before starting with the HA LVM cluster configuration, let us create our logical volumes and volume groups on both the cluster node. Now on node1 and node2 I have /dev/sdc and /dev/sdb as additional storage connected to the nodes respectively.
Create physical volume
[root@node1 ~]# pvcreate /dev/sdc Physical volume "/dev/sdc" successfully created.
Create volume group on /dev/sdc
[root@node1 ~]# vgcreate vgcluster /dev/sdc
Volume group "vgcluster" successfully created
Lastly create a logical volume. here we are creating a logical volume with 400MB size and name as lvcluster on vgcluster volume group
[root@node1 ~]# lvcreate -L 400M -n lvcluster vgcluster
Logical volume "lvcluster" created.
For our demo I will assign XFS filesystem to our lvcluster lvm.
[root@node1 ~]# mkfs.xfs /dev/vgcluster/lvcluster
meta-data=/dev/vgcluster/lvcluster isize=512 agcount=4, agsize=25600 blks
= sectsz=512 attr=2, projid32bit=1
= crc=1 finobt=0, sparse=0
data = bsize=4096 blocks=102400, imaxpct=25
= sunit=0 swidth=0 blks
naming =version 2 bsize=4096 ascii-ci=0 ftype=1
log =internal log bsize=4096 blocks=855, version=2
= sectsz=512 sunit=0 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Validate the changes
[root@node1 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- <15.78g swap centos -wi-ao---- 760.00m lvcluster vgcluster -wi-a----- 400.00m [root@node1 ~]# vgs VG #PV #LV #SN Attr VSize VFree centos 2 2 0 wz--n- <17.52g 1020.00m vgcluster 1 1 0 wz--n- <8.00g <7.61g
Ensure that locking_type is set to 1 as needed, and that lvm2-lvmetad is disabled, by running the following command:
[root@node1 ~]# lvmconf --enable-halvm --services --startstopservices Warning: Stopping lvm2-lvmetad.service, but it can still be activated by: lvm2-lvmetad.socket Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.
On nodes with
lvm2 rpms older than lvm2-2.02.118-2 use the below set of commands
# lvmconf --enable-halvm # systemctl disable lvm2-lvmetad.service ## RHEL 7 # systemctl disable lvm2-lvmetad.socket ## RHEL 7 # systemctl stop lvm2-lvmetad.service ## RHEL 7 # systemctl stop lvm2-lvmetad.socket ## RHEL 7 # chkconfig lvm2-lvmetad off ## RHEL 6 # service lvm2-lvmetad stop ## RHEL 6
There is currently a known issue in which lvm2-lvmetad.socket starts on boot even after it's disabled. To prevent it from starting at all, mask it.
[root@node1 ~]# systemctl mask lvm2-lvmetad.socket Created symlink from /etc/systemd/system/lvm2-lvmetad.socket to /dev/null.
Configure a volume_list in /etc/lvm/lvm.conf that only contains local volume groups that are not shared by cluster nodes. There is no need to include a tag matching the node name as was required for rgmanager configurations.
[root@node1 ~]# vim /etc/lvm/lvm.conf volume_list = [ centos ]
Here centos is the volume group which already exist on my node and which will not be part of my clustered LVM. Hence I have written centos for volume_list in lvm.conf
[root@node1 ~]# vgs
VG #PV #LV #SN Attr VSize VFree
centos 2 2 0 wz--n- <17.52g 1020.00m
vgcluster 1 1 0 wz--n- <8.00g <7.61g
Next rebuild your initramfs
[root@node1 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)
Validate if new initramfs is successfully created.
[root@node1 ~]# ls -l /boot/
total 163452
-rw-r--r-- 1 root root 147859 Sep 26 20:45 config-3.10.0-862.14.4.el7.x86_64
-rw-r--r-- 1 root root 151922 Nov 29 20:23 config-3.10.0-957.1.3.el7.x86_64
drwxr-xr-x. 3 root root 1024 Oct 26 20:34 efi
drwxr-xr-x. 2 root root 1024 Oct 26 20:35 grub
drwx------. 5 root root 1024 Dec 21 11:18 grub2
-rw-------. 1 root root 53692051 Oct 26 20:37 initramfs-0-rescue-43f0510283784c858d39e38e358b4669.img
-rw------- 1 root root 30440965 Dec 26 18:30 initramfs-3.10.0-862.14.4.el7.x86_64.img
-rw------- 1 root root 12604644 Nov 14 19:59 initramfs-3.10.0-862.14.4.el7.x86_64kdump.img
-rw------- 1 root root 30455624 Dec 20 17:51 initramfs-3.10.0-957.1.3.el7.x86_64.img
-rw------- 1 root root 12746618 Dec 21 09:58 initramfs-3.10.0-957.1.3.el7.x86_64kdump.img
-rw-r--r--. 1 root root 611520 Oct 26 20:35 initrd-plymouth.img
drwx------. 2 root root 12288 Oct 26 20:33 lost+found
-rw-r--r-- 1 root root 305158 Sep 26 20:47 symvers-3.10.0-862.14.4.el7.x86_64.gz
-rw-r--r-- 1 root root 314072 Nov 29 20:23 symvers-3.10.0-957.1.3.el7.x86_64.gz
-rw------- 1 root root 3414754 Sep 26 20:45 System.map-3.10.0-862.14.4.el7.x86_64
-rw------- 1 root root 3544010 Nov 29 20:23 System.map-3.10.0-957.1.3.el7.x86_64
-rwxr-xr-x. 1 root root 5877760 Oct 26 20:37 vmlinuz-0-rescue-43f0510283784c858d39e38e358b4669
-rwxr-xr-x 1 root root 6398144 Sep 26 20:45 vmlinuz-3.10.0-862.14.4.el7.x86_64
-rwxr-xr-x 1 root root 6639808 Nov 29 20:23 vmlinuz-3.10.0-957.1.3.el7.x86_64
Lastly reboot the node to activate the changes
Now I will repeat the same steps on node2. Similarly you repeat the same steps on all your cluster nodes.
Use the same specs for the logical volume and volume group as on
node1 or else the cluster  will not work for HA LVM.[root@node2 ~]# pvcreate /dev/sdb Physical volume "/dev/sdb" successfully created. [root@node2 ~]# vgcreate vgcluster /dev/sdb Volume group "vgcluster" successfully created [root@node2 ~]# lvcreate -L 400M -n lvcluster vgcluster Logical volume "lvcluster" created. [root@node2 ~]# mkfs.xfs /dev/vgcluster/lvcluster meta-data=/dev/vgcluster/lvcluster isize=512 agcount=4, agsize=25600 blks = sectsz=512 attr=2, projid32bit=1 = crc=1 finobt=0, sparse=0 data = bsize=4096 blocks=102400, imaxpct=25 = sunit=0 swidth=0 blks naming =version 2 bsize=4096 ascii-ci=0 ftype=1 log =internal log bsize=4096 blocks=855, version=2 = sectsz=512 sunit=0 blks, lazy-count=1 realtime =none extsz=4096 blocks=0, rtextents=0 [root@node2 ~]# vgs VG #PV #LV #SN Attr VSize VFree centos 1 2 0 wz--n- <9.52g 0 vgcluster 1 1 0 wz--n- <8.00g <7.61g [root@node2 ~]# lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert root centos -wi-ao---- <8.78g swap centos -wi-ao---- 760.00m lvcluster vgcluster -wi-a----- 400.00m [root@node2 ~]# lvmconf --enable-halvm --services --startstopservices Warning: Stopping lvm2-lvmetad.service, but it can still be activated by: lvm2-lvmetad.socket Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket. [root@node2 ~]# systemctl mask lvm2-lvmetad.socket Created symlink from /etc/systemd/system/lvm2-lvmetad.socket to /dev/null. [root@node2 ~]# ls -l /boot/ total 87362 -rw-r--r--. 1 root root 140894 Aug 23 2017 config-3.10.0-693.el7.x86_64 drwxr-xr-x. 3 root root 1024 Oct 26 20:34 efi drwxr-xr-x. 2 root root 1024 Oct 26 20:35 grub drwx------. 5 root root 1024 Oct 26 20:38 grub2 -rw-------. 1 root root 53692051 Oct 26 20:37 initramfs-0-rescue-43f0510283784c858d39e38e358b4669.img -rw------- 1 root root 19706468 Dec 26 18:25 initramfs-3.10.0-693.el7.x86_64.img -rw-r--r--. 1 root root 611520 Oct 26 20:35 initrd-plymouth.img drwx------. 2 root root 12288 Oct 26 20:33 lost+found -rw-r--r--. 1 root root 293027 Aug 23 2017 symvers-3.10.0-693.el7.x86_64.gz -rw-------. 1 root root 3228420 Aug 23 2017 System.map-3.10.0-693.el7.x86_64 -rwxr-xr-x. 1 root root 5877760 Oct 26 20:37 vmlinuz-0-rescue-43f0510283784c858d39e38e358b4669 -rwxr-xr-x. 1 root root 5877760 Aug 23 2017 vmlinuz-3.10.0-693.el7.x86_64
Configure HA LVM on cluster nodes
Now our environment is all set up to start configuring HA LVM. First of all we need to create a LVM resource group
Create LVM resource group for LVM
Now we can create the HA LVM resources. Where LVM is a generic resource that we are going to use in the cluster, and halvm is just the name that we are assigning. volgrpname equals vgcluster, exclusive is true, and we are going to put it in a group with name halvmfs.
[root@node1 ~]# pcs resource create halvm LVM volgrpname=vgcluster exclusive=true --group halvmfs Assumed agent name 'ocf:heartbeat:LVM' (deduced from 'LVM')
Now that we have created the resource for the cluster, we can verify that it has indeed been started.
[root@node1 ~]# pcs status Cluster name: mycluster Stack: corosync Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Wed Dec 26 18:34:28 2018 Last change: Wed Dec 26 18:34:14 2018 by root via cibadmin on node1.example.com 2 nodes configured 5 resources configured Online: [ node1.example.com node2.example.com ] Full list of resources: Resource Group: apache-group apache-ip (ocf::heartbeat:IPaddr2): Started node2.example.com apache-service (ocf::heartbeat:apache): Started node2.example.com Resource Group: ftp-group ftp-ip (ocf::heartbeat:IPaddr2): Stopped ftp-service (systemd:vsftpd): Stopped Resource Group: halvmfs halvm (ocf::heartbeat:LVM): Started node2.example.com Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
As you see our HA LVM service has started successfully on node2.example.com. Now we need to take care of the file system, and the file system needs to be mounted somewhere.
Create resource group to mount the file system
For the sake of this article to configure HA LVM I will create /xfs directory as my mount point on both my cluster nodes.
[root@node1 ~]# mkdir /xfs [root@node2 ~]# mkdir /xfs
Next we need to create a resource that mounts the file system through the cluster.
[root@node1 ~]# pcs resource create xfsfs Filesystem device="/dev/vgcluster/lvcluster" directory="/xfs" fstype="xfs" --group halvmfs
Here we are creating a resource with Filesystem type for our logical volume /dev/vgcluster/lvcluster which should be mounted on /xfs and let it be part of our existing halvmfs group
So our last command execution was successful so let us validate the pcs cluster status.
[root@node1 ~]# pcs status Cluster name: mycluster Stack: corosync Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Wed Dec 26 18:37:51 2018 Last change: Wed Dec 26 18:37:29 2018 by root via cibadmin on node1.example.com 2 nodes configured 6 resources configured Online: [ node1.example.com node2.example.com ] Full list of resources: Resource Group: apache-group apache-ip (ocf::heartbeat:IPaddr2): Started node2.example.com apache-service (ocf::heartbeat:apache): Started node2.example.com Resource Group: ftp-group ftp-ip (ocf::heartbeat:IPaddr2): Stopped ftp-service (systemd:vsftpd): Stopped Resource Group: halvmfs halvm (ocf::heartbeat:LVM): Started node2.example.com xfsfs (ocf::heartbeat:Filesystem): Started node2.example.com Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
As you see our resource xfsfs has been successfully started on node2.example.com
Validate HA LVM configuration
Now since we know our HA cluster resource is running on node2.example.com, validate if the logical volume is successfully mounted on /xfs
[root@node2 ~]# mount | grep xfs /dev/mapper/vgcluster-lvcluster on /xfs type xfs (rw,relatime,attr2,inode64,noquota)
So all looks good and working as expected.
Validate HA LVM failover
Now let us do the validation to make sure failover works for our HA LVM cluster resource. So for this purpose we will change node2.example.com state to standby
[root@node2 ~]# pcs cluster standby node2.example.com
Now validate the pcs cluster status
[root@node2 ~]# pcs status Cluster name: mycluster Stack: corosync Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Wed Dec 26 18:39:56 2018 Last change: Wed Dec 26 18:39:32 2018 by root via cibadmin on node2.example.com 2 nodes configured 6 resources configured Node node2.example.com: standby Online: [ node1.example.com ] Full list of resources: Resource Group: apache-group apache-ip (ocf::heartbeat:IPaddr2): Started node1.example.com apache-service (ocf::heartbeat:apache): Started node1.example.com Resource Group: ftp-group ftp-ip (ocf::heartbeat:IPaddr2): Stopped ftp-service (systemd:vsftpd): Stopped Resource Group: halvmfs halvm (ocf::heartbeat:LVM): Started node1.example.com xfsfs (ocf::heartbeat:Filesystem): Started node1.example.com Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
As we see now our HA LVM cluster resource is started on node1.example.com since node2 is on standby
Next check if the logical volume lvcluster is mounted on node1
[root@node1 ~]# mount | grep xfs /dev/mapper/vgcluster-lvcluster on /xfs type xfs (rw,relatime,attr2,inode64,noquota)
So our failover is also working.
Lastly I hope the steps from the article to configure HA LVM on a Cluster in Linux was helpful. So, let me know your suggestions and feedback using the comment section.
Hi, thanks for great article, trying to setup this configuration, I get same results like Kevin, so I think the mount on failover takes the local logical volumes and not the remote file system, so the files created on one node is never seeing on the another one, the explanation may be lack some variable ?, the mount of resources is ok on both nodes. Please can you clarify little more.
Hi it works, but what happen if i copy the file to that lvm then suddently the node1 down?
Do you mean in the middle of copy process, the node1 goes down?
Yes correct, is there any solution? i tried it yesterday, the copy process is failed.
maybe with the virtual IP but currently i have no idea
This can be tricky, hard to predict the behavior but theoretically on paper the copied file will be in corrupted state as unless it is synced to the storage it cannot be protected.
Hi,
Thats very nice manual. However it will be better with stonith sbd + softdog.
Thanks a lot for your times and jobs.
May I use this for centralizing storage, I mean, for example, have 4 nodes, 2 for storage and 2 for serving a website, so the website folder with the source will be stored on the nodes for storage, and mounted as NFS or any other method on ondes for serving the website.
Yes with HA LVM cluster that can be done.
Hi, nice good guide.
My question is that I created test.txt in node1, then make node1 as standby and failover to node2, the LV will mount on node2 successfully, I suppose I can see the test.txt in node2 now, but the file is not showing in node2.
Thanks for your feedback. If the file test.txt is created on the shared filesystem then yes it must appear on node2 after failover
first of all awesome documentation. Thanks.
My query is do we need to rebuild initrd on node 2 ?
Thank you for your feedback.
Yes, you need to follow and repeat the same set of steps as done on node1 on the remaining nodes of the cluster.