This is a two part article, here I will share the steps to configure OpenStack High Availability (HA) between two controllers. In the second part I will share the steps to configure HAProxy and move keystone service endpoints to loadbalancer. By default if your bring up a controller and compute node using tripleo configuration then the controllers will by default get configured via pacemaker cluster. But if you are manually bringing up your openstack setup using packstack or devstack or by manually creating all the database and services then you will have to manually configure cluster between the controllers to configure OpenStack High Availability (HA).
Configure OpenStack High Availability (HA)
For the sake of this article I brought up two controller nodes using packstack on two different virtual machines using CentOS 7 running on Oracle VirtualBox installed on my Linux Server. After the successful completion of packstack you will observe keystonerc_admin file in the home folder of the root user.
Installing the Pacemaker resource manager
Since we will configure OpenStack High Availability using pacemaker and corosync, first of all we need to install all the rpms required for the cluster setup. So we will install Pacemaker to manage the VIPs that we will use with HAProxy to make the web services highly available.
So install pacemaker on all the controller nodes
[root@controller2 ~]# yum install -y pcs fence-agents-all [root@controller1 ~]# yum install -y pcs fence-agents-all
Verify that the software installed correctly by running the following command:
[root@controller1 ~]# rpm -q pcs pcs-0.9.162-5.el7.centos.2.x86_64 [root@controller2 ~]# rpm -q pcs pcs-0.9.162-5.el7.centos.2.x86_64
Next, add rules to the firewall to allow cluster traffic:
[root@controller1 ~]# firewall-cmd --permanent --add-service=high-availability success [root@controller1 ~]# firewall-cmd --reload success [root@controller2 ~]# firewall-cmd --permanent --add-service=high-availability success [root@controller2 ~]# firewall-cmd --reload success
If you run into any problems during testing, you might want to disable the firewall and SELinux entirely until you have everything working. This may create significant security issues and should not be performed on machines that will be exposed to the outside world, but may be appropriate during development and testing on a protected host.
The installed packages will create a hacluster user with a disabled password. While this is fine for running pcs commands locally, the account needs a login password in order to perform such tasks as syncing the corosync configuration, or starting and stopping the cluster on other nodes.
Set the password for the Pacemaker cluster on each controller node using the following command:
[root@controller1 ~]# passwd hacluster Changing password for user hacluster. New password: Retype new password: passwd: all authentication tokens updated successfully. [root@controller2 ~]# passwd hacluster Changing password for user hacluster. New password: Retype new password: passwd: all authentication tokens updated successfully.
Start the Pacemaker cluster manager on each node:
[root@controller1 ~]# systemctl start pcsd.service [root@controller1 ~]# systemctl enable pcsd.service Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service. [root@controller2 ~]# systemctl start pcsd.service [root@controller2 ~]# systemctl enable pcsd.service Created symlink from /etc/systemd/system/multi-user.target.wants/pcsd.service to /usr/lib/systemd/system/pcsd.service.
Configure Corosync
To configure Openstack High Availability we need to configure corosync on both the nodes, use pcs cluster auth to authenticate as the hacluster user:
[root@controller1 ~]# pcs cluster auth controller1 controller2 Username: hacluster Password: controller2: Authorized controller1: Authorized [root@controller2 ~]# pcs cluster auth controller1 controller2 Username: hacluster Password: controller2: Authorized controller1: Authorized
Finally, run the following commands on the first node to create the cluster and start it. Here our cluster name will be openstack
[root@controller1 ~]# pcs cluster setup --start --name openstack controller1 controller2
Destroying cluster on nodes: controller1, controller2...
controller1: Stopping Cluster (pacemaker)...
controller2: Stopping Cluster (pacemaker)...
controller1: Successfully destroyed cluster
controller2: Successfully destroyed cluster
Sending 'pacemaker_remote authkey' to 'controller1', 'controller2'
controller1: successful distribution of the file 'pacemaker_remote authkey'
controller2: successful distribution of the file 'pacemaker_remote authkey'
Sending cluster config files to the nodes...
controller1: Succeeded
controller2: Succeeded
Starting cluster on nodes: controller1, controller2...
controller1: Starting Cluster...
controller2: Starting Cluster...
Synchronizing pcsd certificates on nodes controller1, controller2...
controller2: Success
controller1: Success
Restarting pcsd on the nodes in order to reload the certificates...
controller2: Success
controller1: Success
Enable the pacemaker and corosync services on both the controller so they can automatically start on boot
[root@controller1 ~]# systemctl enable pacemaker Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /usr/lib/systemd/system/pacemaker.service. [root@controller1 ~]# systemctl enable corosync Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /usr/lib/systemd/system/corosync.service. [root@controller2 ~]# systemctl enable corosync Created symlink from /etc/systemd/system/multi-user.target.wants/corosync.service to /usr/lib/systemd/system/corosync.service. [root@controller2 ~]# systemctl enable pacemaker Created symlink from /etc/systemd/system/multi-user.target.wants/pacemaker.service to /usr/lib/systemd/system/pacemaker.service.
Validate cluster using pacemaker
Verify that the cluster started successfully using the following command on both the nodes:
[root@controller1 ~]# pcs status Cluster name: openstack Stack: corosync Current DC: controller2 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Tue Oct 16 11:51:13 2018 Last change: Tue Oct 16 11:50:51 2018 by root via cibadmin on controller1 2 nodes configured 0 resources configured Online: [ controller1 controller2 ] No resources Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
[root@controller2 ~]# pcs status Cluster name: openstack WARNING: no stonith devices and stonith-enabled is not false Stack: corosync Current DC: controller2 (version 1.1.18-11.el7_5.3-2b07d5c5a9) - partition with quorum Last updated: Mon Oct 15 17:04:29 2018 Last change: Mon Oct 15 16:49:09 2018 by hacluster via crmd on controller2 2 nodes configured 0 resources configured Online: [ controller1 controller2 ] No resources Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled
How to start the Cluster
Now that corosync is configured, it is time to start the cluster. The command below will start corosync and pacemaker on both nodes in the cluster. If you are issuing the start command from a different node than the one you ran the pcs cluster auth command on earlier, you must authenticate on the current node you are logged into before you will be allowed to start the cluster.
[root@controller1 ~]# pcs cluster start --all
An alternative to using the pcs cluster start --all command is to issue either of the below command sequences on each node in the cluster separately:
[root@controller1 ~]# pcs cluster start
Starting Cluster...
or
[root@controller1 ~]# systemctl start corosync.service [root@controller1 ~]# systemctl start pacemaker.service
Verify Corosync Installation
First, use corosync-cfgtool to check whether cluster communication is happy:
[root@controller2 ~]# corosync-cfgtool -s
Printing ring status.
Local node ID 2
RING ID 0
id = 192.168.122.22
status = ring 0 active with no faults
So all looks normal with our fixed IP address (not a 127.0.0.x loopback address) listed as the id, and no faults for the status.
If you see something different, you might want to start by checking the node’s network, firewall and SELinux configurations.
Next, check the membership and quorum APIs:
[root@controller2 ~]# corosync-cmapctl | grep members
runtime.totem.pg.mrp.srp.members.1.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.1.ip (str) = r(0) ip(192.168.122.20)
runtime.totem.pg.mrp.srp.members.1.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.1.status (str) = joined
runtime.totem.pg.mrp.srp.members.2.config_version (u64) = 0
runtime.totem.pg.mrp.srp.members.2.ip (str) = r(0) ip(192.168.122.22)
runtime.totem.pg.mrp.srp.members.2.join_count (u32) = 1
runtime.totem.pg.mrp.srp.members.2.status (str) = joined
Check the status of corosync service
[root@controller2 ~]# pcs status corosync
Membership information
----------------------
Nodeid Votes Name
1 1 controller1
2 1 controller2 (local)
You should see both nodes have joined the cluster.
Repeat the same steps on both the controller to validate the corosync services
Verify the cluster configuration
Before we make any changes, it’s a good idea to check the validity of the configuration.
[root@controller1 ~]# crm_verify -L -V error: unpack_resources: Resource start-up disabled since no STONITH resources have been defined error: unpack_resources: Either configure some or disable STONITH with the stonith-enabled option error: unpack_resources: NOTE: Clusters with shared data need STONITH to ensure data integrity Errors found during check: config not valid
As you can see, the tool has found some errors.
In order to guarantee the safety of your data, [5] fencing (also called STONITH) is enabled by default. However, it also knows when no STONITH configuration has been supplied and reports this as a problem (since the cluster will not be able to make progress if a situation requiring node fencing arises).
We will disable this feature for now and configure it later. To disable STONITH, set the stonith-enabled cluster option to false on both the controller nodes:
[root@controller1 ~]# pcs property set stonith-enabled=false [root@controller1 ~]# crm_verify -L [root@controller2 ~]# pcs property set stonith-enabled=false [root@controller2 ~]# crm_verify -L With the new cluster option set, the configuration is now valid.
stonith-enabled=false is completely inappropriate for a production cluster. It tells the cluster to simply pretend that the nodes which fails are safely in powered off state. Some vendors will refuse to support clusters that have STONITH disabled.
I will continue this article i.e to configure OpenStack High Availability in separate part. In the next part I will share the steps to configure HAProxy and we will manage it as a resource. Also the detail steps to move OpenStack API endpoints behind the cluster load balancer.
