How to configure High Availability LVM i.e. HA LVM on CentOS or RHEL 7 Linux without GFS2. How can I configure my cluster service or resource to activate and monitor shared LVM volumes? Uncontrolled simultaneous access to shared storage can lead to data corruption. Storage access must be managed like any other active/passive service – it must only be active on a single machine at a time. Is there a way to configure LVM volumes for active/passive use in a High Availability cluster?

How to configure HA LVM cluster resource to share LVM in Linux

To configure HA LVM, as the name suggests you need logical volumes. So make sure you have logical volumes and volume groups with the same name on all your nodes of the cluster.

Earlier I had shared an article about Cluster Architecture and Types of Clusters and also step by step guide to configure HA Cluster with three nodes. Now later I had to remove one of the cluster node to demonstrate you about two node cluster setup and it’s configuration.

So I will continue to use the same setup for demonstration of this article. As you see I already have some resource groups which are part of my cluster. I have written another article to help you understand all about resource groups and resource constraints on Cluster.

[root@node1 ~]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Wed Dec 26 18:37:51 2018
Last change: Wed Dec 26 18:37:29 2018 by root via cibadmin on node1.example.com

2 nodes configured
6 resources configured

Online: [ node1.example.com node2.example.com ]

Full list of resources:

 Resource Group: apache-group
     apache-ip  (ocf::heartbeat:IPaddr2):       Started node2.example.com
     apache-service     (ocf::heartbeat:apache):        Started node2.example.com
 Resource Group: ftp-group
     ftp-ip     (ocf::heartbeat:IPaddr2):       Stopped
     ftp-service        (systemd:vsftpd):       Stopped

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

 

Setting up the environment

Now before starting with the HA LVM cluster configuration, let us create our logical volumes and volume groups on both the cluster node. Now on node1 and node2 I have /dev/sdc and /dev/sdb as additional storage connected to the nodes respectively.

Create physical volume

[root@node1 ~]# pvcreate /dev/sdc
  Physical volume "/dev/sdc" successfully created.

Create volume group on /dev/sdc

[root@node1 ~]# vgcreate vgcluster /dev/sdc
  Volume group "vgcluster" successfully created

Lastly create a logical volume. here we are creating a logical volume with 400MB size and name as lvcluster on vgcluster volume group

[root@node1 ~]# lvcreate -L 400M -n lvcluster vgcluster
  Logical volume "lvcluster" created.

For our demo I will assign XFS filesystem to our lvcluster lvm.

[root@node1 ~]# mkfs.xfs /dev/vgcluster/lvcluster
meta-data=/dev/vgcluster/lvcluster isize=512    agcount=4, agsize=25600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=102400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

Validate the changes

[root@node1 ~]# lvs
  LV        VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root      centos    -wi-ao---- <15.78g
  swap      centos    -wi-ao---- 760.00m
  lvcluster vgcluster -wi-a----- 400.00m

[root@node1 ~]# vgs
  VG        #PV #LV #SN Attr   VSize   VFree
  centos      2   2   0 wz--n- <17.52g 1020.00m
  vgcluster   1   1   0 wz--n-  <8.00g   <7.61g

Ensure that locking_type is set to 1 as needed, and that lvm2-lvmetad is disabled, by running the following command:

[root@node1 ~]# lvmconf --enable-halvm --services --startstopservices
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.
NOTE:
On nodes with lvm2 rpms older than lvm2-2.02.118-2 use the below set of commands
# lvmconf --enable-halvm
# systemctl disable lvm2-lvmetad.service   ## RHEL 7
# systemctl disable lvm2-lvmetad.socket    ## RHEL 7
# systemctl stop lvm2-lvmetad.service      ## RHEL 7
# systemctl stop lvm2-lvmetad.socket       ## RHEL 7
# chkconfig lvm2-lvmetad off               ## RHEL 6
# service lvm2-lvmetad stop                ## RHEL 6

There is currently a known issue in which lvm2-lvmetad.socket starts on boot even after it’s disabled. To prevent it from starting at all, mask it.

[root@node1 ~]# systemctl mask lvm2-lvmetad.socket
Created symlink from /etc/systemd/system/lvm2-lvmetad.socket to /dev/null.

Configure a volume_list in /etc/lvm/lvm.conf that only contains local volume groups that are not shared by cluster nodes. There is no need to include a tag matching the node name as was required for rgmanager configurations.

[root@node1 ~]# vim /etc/lvm/lvm.conf
volume_list = [ centos ]

Here centos is the volume group which already exist on my node and which will not be part of my clustered LVM. Hence I have written centos for volume_list in lvm.conf

[root@node1 ~]# vgs
  VG        #PV #LV #SN Attr   VSize   VFree
  centos      2   2   0 wz--n- <17.52g 1020.00m
  vgcluster   1   1   0 wz--n-  <8.00g   <7.61g

Next rebuild your initramfs

[root@node1 ~]# dracut -H -f /boot/initramfs-$(uname -r).img $(uname -r)

Validate if new initramfs is successfully created.

[root@node1 ~]# ls -l /boot/
total 163452
-rw-r--r--  1 root root   147859 Sep 26 20:45 config-3.10.0-862.14.4.el7.x86_64
-rw-r--r--  1 root root   151922 Nov 29 20:23 config-3.10.0-957.1.3.el7.x86_64
drwxr-xr-x. 3 root root     1024 Oct 26 20:34 efi
drwxr-xr-x. 2 root root     1024 Oct 26 20:35 grub
drwx------. 5 root root     1024 Dec 21 11:18 grub2
-rw-------. 1 root root 53692051 Oct 26 20:37 initramfs-0-rescue-43f0510283784c858d39e38e358b4669.img
-rw-------  1 root root 30440965 Dec 26 18:30 initramfs-3.10.0-862.14.4.el7.x86_64.img
-rw-------  1 root root 12604644 Nov 14 19:59 initramfs-3.10.0-862.14.4.el7.x86_64kdump.img
-rw-------  1 root root 30455624 Dec 20 17:51 initramfs-3.10.0-957.1.3.el7.x86_64.img
-rw-------  1 root root 12746618 Dec 21 09:58 initramfs-3.10.0-957.1.3.el7.x86_64kdump.img
-rw-r--r--. 1 root root   611520 Oct 26 20:35 initrd-plymouth.img
drwx------. 2 root root    12288 Oct 26 20:33 lost+found
-rw-r--r--  1 root root   305158 Sep 26 20:47 symvers-3.10.0-862.14.4.el7.x86_64.gz
-rw-r--r--  1 root root   314072 Nov 29 20:23 symvers-3.10.0-957.1.3.el7.x86_64.gz
-rw-------  1 root root  3414754 Sep 26 20:45 System.map-3.10.0-862.14.4.el7.x86_64
-rw-------  1 root root  3544010 Nov 29 20:23 System.map-3.10.0-957.1.3.el7.x86_64
-rwxr-xr-x. 1 root root  5877760 Oct 26 20:37 vmlinuz-0-rescue-43f0510283784c858d39e38e358b4669
-rwxr-xr-x  1 root root  6398144 Sep 26 20:45 vmlinuz-3.10.0-862.14.4.el7.x86_64
-rwxr-xr-x  1 root root  6639808 Nov 29 20:23 vmlinuz-3.10.0-957.1.3.el7.x86_64

 

Lastly reboot the node to activate the changes

Now I will repeat the same steps on node2. Similarly you repeat the same steps on all your cluster nodes.

IMPORTANT NOTE:
Use the same specs for the logical volume and volume group as on node1 or else the cluster  will not work for HA LVM.
[root@node2 ~]# pvcreate /dev/sdb
  Physical volume "/dev/sdb" successfully created.

[root@node2 ~]# vgcreate vgcluster /dev/sdb
  Volume group "vgcluster" successfully created

[root@node2 ~]# lvcreate -L 400M -n lvcluster vgcluster
  Logical volume "lvcluster" created.

[root@node2 ~]#  mkfs.xfs /dev/vgcluster/lvcluster
meta-data=/dev/vgcluster/lvcluster isize=512    agcount=4, agsize=25600 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=0, sparse=0
data     =                       bsize=4096   blocks=102400, imaxpct=25
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0 ftype=1
log      =internal log           bsize=4096   blocks=855, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

[root@node2 ~]# vgs
  VG        #PV #LV #SN Attr   VSize  VFree
  centos      1   2   0 wz--n- <9.52g     0
  vgcluster   1   1   0 wz--n- <8.00g <7.61g

[root@node2 ~]# lvs
  LV        VG        Attr       LSize   Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  root      centos    -wi-ao----  <8.78g
  swap      centos    -wi-ao---- 760.00m
  lvcluster vgcluster -wi-a----- 400.00m

[root@node2 ~]# lvmconf --enable-halvm --services --startstopservices
Warning: Stopping lvm2-lvmetad.service, but it can still be activated by:
  lvm2-lvmetad.socket
Removed symlink /etc/systemd/system/sysinit.target.wants/lvm2-lvmetad.socket.

[root@node2 ~]# systemctl mask lvm2-lvmetad.socket
Created symlink from /etc/systemd/system/lvm2-lvmetad.socket to /dev/null.

[root@node2 ~]# ls -l /boot/
total 87362
-rw-r--r--. 1 root root   140894 Aug 23  2017 config-3.10.0-693.el7.x86_64
drwxr-xr-x. 3 root root     1024 Oct 26 20:34 efi
drwxr-xr-x. 2 root root     1024 Oct 26 20:35 grub
drwx------. 5 root root     1024 Oct 26 20:38 grub2
-rw-------. 1 root root 53692051 Oct 26 20:37 initramfs-0-rescue-43f0510283784c858d39e38e358b4669.img
-rw-------  1 root root 19706468 Dec 26 18:25 initramfs-3.10.0-693.el7.x86_64.img
-rw-r--r--. 1 root root   611520 Oct 26 20:35 initrd-plymouth.img
drwx------. 2 root root    12288 Oct 26 20:33 lost+found
-rw-r--r--. 1 root root   293027 Aug 23  2017 symvers-3.10.0-693.el7.x86_64.gz
-rw-------. 1 root root  3228420 Aug 23  2017 System.map-3.10.0-693.el7.x86_64
-rwxr-xr-x. 1 root root  5877760 Oct 26 20:37 vmlinuz-0-rescue-43f0510283784c858d39e38e358b4669
-rwxr-xr-x. 1 root root  5877760 Aug 23  2017 vmlinuz-3.10.0-693.el7.x86_64

 

Configure HA LVM on cluster nodes

Now our environment is all set up to start configuring HA LVM. First of all we need to create a LVM resource group

 

Create LVM resource group for LVM

Now we can create the HA LVM resources. Where LVM is a generic resource that we are going to use in the cluster, and halvm is just the name that we are assigning. volgrpname equals vgcluster, exclusive is true, and we are going to put it in a group with name halvmfs.

[root@node1 ~]# pcs resource create halvm LVM volgrpname=vgcluster exclusive=true --group halvmfs
Assumed agent name 'ocf:heartbeat:LVM' (deduced from 'LVM')

Now that we have created the resource for the cluster, we can verify that it has indeed been started.

[root@node1 ~]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Wed Dec 26 18:34:28 2018
Last change: Wed Dec 26 18:34:14 2018 by root via cibadmin on node1.example.com

2 nodes configured
5 resources configured

Online: [ node1.example.com node2.example.com ]

Full list of resources:

 Resource Group: apache-group
     apache-ip  (ocf::heartbeat:IPaddr2):       Started node2.example.com
     apache-service     (ocf::heartbeat:apache):        Started node2.example.com
 Resource Group: ftp-group
     ftp-ip     (ocf::heartbeat:IPaddr2):       Stopped
     ftp-service        (systemd:vsftpd):       Stopped
 Resource Group: halvmfs
     halvm      (ocf::heartbeat:LVM):   Started node2.example.com

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

As you see our HA LVM service has started successfully on node2.example.com. Now we need to take care of the file system, and the file system needs to be mounted somewhere.

 

Create resource group to mount the file system

For the sake of this article to configure HA LVM I will create /xfs directory as my mount point on both my cluster nodes.

[root@node1 ~]# mkdir /xfs
[root@node2 ~]# mkdir /xfs

Next we need to create a resource that mounts the file system through the cluster.

[root@node1 ~]# pcs resource create xfsfs Filesystem device="/dev/vgcluster/lvcluster" directory="/xfs" fstype="xfs" --group halvmfs

Here we are creating a resource with Filesystem type for our logical volume /dev/vgcluster/lvcluster which should be mounted on /xfs and let it be part of our existing halvmfs group

 

So our last command execution was successful so let us validate the pcs cluster status.

[root@node1 ~]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Wed Dec 26 18:37:51 2018
Last change: Wed Dec 26 18:37:29 2018 by root via cibadmin on node1.example.com

2 nodes configured
6 resources configured

Online: [ node1.example.com node2.example.com ]

Full list of resources:

 Resource Group: apache-group
     apache-ip  (ocf::heartbeat:IPaddr2):       Started node2.example.com
     apache-service     (ocf::heartbeat:apache):        Started node2.example.com
 Resource Group: ftp-group
     ftp-ip     (ocf::heartbeat:IPaddr2):       Stopped
     ftp-service        (systemd:vsftpd):       Stopped
 Resource Group: halvmfs
     halvm      (ocf::heartbeat:LVM):   Started node2.example.com
     xfsfs      (ocf::heartbeat:Filesystem):    Started node2.example.com


Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

As you see our resource xfsfs has been successfully started on node2.example.com

 

Validate HA LVM configuration

Now since we know our HA cluster resource is running on node2.example.com, validate if the logical volume is successfully mounted on /xfs

[root@node2 ~]# mount | grep xfs
/dev/mapper/vgcluster-lvcluster on /xfs type xfs (rw,relatime,attr2,inode64,noquota)

So all looks good and working as expected.

 

Validate HA LVM failover

Now let us do the validation to make sure failover works for our HA LVM cluster resource. So for this purpose we will change node2.example.com state to standby

[root@node2 ~]# pcs cluster standby node2.example.com

Now validate the pcs cluster status

[root@node2 ~]# pcs status
Cluster name: mycluster
Stack: corosync
Current DC: node1.example.com (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum
Last updated: Wed Dec 26 18:39:56 2018
Last change: Wed Dec 26 18:39:32 2018 by root via cibadmin on node2.example.com

2 nodes configured
6 resources configured

Node node2.example.com: standby
Online: [ node1.example.com ]

Full list of resources:

 Resource Group: apache-group
     apache-ip  (ocf::heartbeat:IPaddr2):       Started node1.example.com
     apache-service     (ocf::heartbeat:apache):        Started node1.example.com
 Resource Group: ftp-group
     ftp-ip     (ocf::heartbeat:IPaddr2):       Stopped
     ftp-service        (systemd:vsftpd):       Stopped
 Resource Group: halvmfs
     halvm      (ocf::heartbeat:LVM):   Started node1.example.com
     xfsfs      (ocf::heartbeat:Filesystem):    Started node1.example.com

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

As we see now our HA LVM cluster resource is started on node1.example.com since node2 is on standby

Next check if the logical volume lvcluster is mounted on node1

[root@node1 ~]# mount | grep xfs
/dev/mapper/vgcluster-lvcluster on /xfs type xfs (rw,relatime,attr2,inode64,noquota)

So our failover is also working.

 

Lastly I hope the steps from the article to configure HA LVM on a Cluster in Linux was helpful. So, let me know your suggestions and feedback using the comment section.

2 Comments

    1. Thank you for your feedback.

      Yes, you need to follow and repeat the same set of steps as done on node1 on the remaining nodes of the cluster.

Leave a Reply

Your email address will not be published. Required fields are marked *