Install & configure glusterfs replicated volume RHEL/CentOS 8


Storage

This is a multi-part article where in a series of article we will learn about Gluster File System in Linux, below are the topics we will cover:

 

Lab Environment

I have create two Virtual Machines using Oracle VirtualBox which is installed on a Linux Server. These two VMs are installed with CentOS 8. Below are the configuration spec of these two virtual machines:

Configuration Node 1 Node 2 Node3
Hostname/FQDN glusterfs-1.example.com glusterfs-2.example.com glusterfs-3.example.com
OS CentOS 8 CentOS 8 CentOS 8
IP Address 10.10.10.6 10.10.10.12 10.10.10.13
Storage 1 (/dev/sda) 20GB 20GB 20GB
Storage 2 (/dev/sdb) 10GB 10GB 10GB

 

Name Resolution

You must configure DNS to resolve hostname or alternatively use /etc/hosts file. I have updated /etc/hosts file with the IPs of my GlusterFS nodes

# cat /etc/hosts
127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4
::1         localhost localhost.localdomain localhost6 localhost6.localdomain6
10.10.10.6   glusterfs-1     glusterfs-1.example.com
10.10.10.12   glusterfs-2     glusterfs-2.example.com
10.10.10.13   glusterfs-3     glusterfs-3.example.com

 

Install Gluster File system

Install GlusterFS on CentOS 8

Depending upon your environment you can download the repo file of glusterfs from the official page. I am using internal network on my CentOS 8 virtual machine so there is no internet connectivity in my VM which is why I downloaded the glusterfs repo on one of my RHEL 8 node and then created an offline repo by downloading the entire repository

NOTE:
GlusterFS repo has been removed and now you can manually download the required rpom from CentOS official repository

 

Enable PowerTools repo

You must also enable PowerTools repo or else you will get below error while installing glusterfs-server

python3-pyxattr is needed by glusterfs-srver which is provded by powertools repo from centOS 8 so this also needs to be enabled

To enable powertools you can manually enable the repo using "enabled=1" param in /etc/yum.repos.d/CentOS-PowerTools.repo or you can install yum-utils first

[root@glusterfs-1 ~]# yum -y install yum-utils

and then later using yum-config-manager you can enable the PowerTools repo

[root@glusterfs-1 ~]# yum-config-manager --enable PowerTools

To list the available repos

[root@glusterfs-1 ~]# yum repolist
CentOS-8 - AppStream                                             5.1 kB/s | 4.3 kB     00:00
CentOS-8 - Base                                                  6.1 kB/s | 3.8 kB     00:00
CentOS-8 - Extras                                                256  B/s | 1.5 kB     00:06
CentOS-8 - PowerTools                                            815 kB/s | 2.0 MB     00:02
Extra Packages for Enterprise Linux 8 - x86_64                   6.1 kB/s | 7.7 kB     00:01
GlusterFS clustered file-system                                  2.9 MB/s | 3.0 kB     00:00
repo id                   repo name                                                   status
AppStream                 CentOS-8 - AppStream                                         5,001
BaseOS                    CentOS-8 - Base                                              1,784
PowerTools                CentOS-8 - PowerTools                                        1,499
epel                      Extra Packages for Enterprise Linux 8 - x86_64               4,541
extras                    CentOS-8 - Extras                                                3

You can install glusterfs-server to install GlusterFS

[root@glusterfs-1 ~]# rpm -Uvh https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-9/Packages/g/glusterfs-server-9.4-1.el8.x86_64.rpm

 

Install GlusterFS on Red Hat 8 (RHEL 8)

There are various source and methods to install GlusterFS in RHEL 8

Next to install Red Hat Gluster Storage using redhat-storage-server rpm

# yum install redhat-storage-server

 

Start glusterd service

Next start the glusterd service on both the cluster nodes

[root@glusterfs-1 ~]# systemctl start glusterd

Verify the status of the service and make sure it is in active running state:

[root@glusterfs-1 ~]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; disabled; vendor preset: disabled)
   Active: active (running) since Sun 2020-01-26 02:19:31 IST; 4s ago
     Docs: man:glusterd(8)
  Process: 2855 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCCESS)
 Main PID: 2856 (glusterd)
    Tasks: 9 (limit: 26213)
   Memory: 3.9M
   CGroup: /system.slice/glusterd.service
           └─2856 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Jan 26 02:19:31 glusterfs-1.example.com systemd[1]: Starting GlusterFS, a clustered file-system server...
Jan 26 02:19:31 glusterfs-1.example.com systemd[1]: Started GlusterFS, a clustered file-system server.

Enable the service so that the service comes up automatically

[root@glusterfs-1 ~]# systemctl enable glusterd

 

Create Partition

If you already have an additional logical volume for Gluster File System then you can ignore these steps.
We will create a new logical volume on both our CentOS 8 nodes to create a GlusterFS replicated volume. Now since I have already explained the steps required to create a partition, I won't explain these commands again here.

[root@glusterfs-1 ~]# pvcreate /dev/sdb  <-- To create physical volume
[root@glusterfs-1 ~]# vgextend rhel /dev/sdb  <-- Extend my existing volume group
[root@glusterfs-1 ~]# lvcreate -L 2G -n brick1 rhel  <-- Create logical volume named "brick1" with size 2GB using rhel VG
[root@glusterfs-1 ~]# mkfs.xfs /dev/mapper/rhel-brick1  <-- Format the logical volume using XFS File System

Create a mount point for this new logical volume

[root@glusterfs-3 ~]# mkdir -p /bricks/brick1

Mount the logical volume on this mount point

[root@glusterfs-3 ~]# mount /dev/mapper/rhel-brick1 /bricks/brick1

Verify the same

[root@glusterfs-1 ~]# df -Th /bricks/brick1
Filesystem              Type  Size  Used Avail Use% Mounted on
/dev/mapper/rhel-brick1 xfs   2.0G   47M  2.0G   3% /bricks/brick1

Similarly we will create /dev/mapper/rhel-brick2 on gcluster-2 and /dev/mapper/rhel-brick3 on gcluster-3 node

[root@glusterfs-2 ~]# df -Th /bricks/brick2
Filesystem              Type  Size  Used Avail Use% Mounted on
/dev/mapper/rhel-brick2 xfs   2.0G   47M  2.0G   3% /bricks/brick2

[root@glusterfs-3 ~]# df -Th /bricks/brick3
Filesystem              Type  Size  Used Avail Use% Mounted on
/dev/mapper/rhel-brick3 xfs   2.0G   47M  2.0G   3% /bricks/brick3
NOTE:
Update /etc/fstab of these logical volume on the respective cluster nodes to make sure these gluster file systems gets mounted post reboot

 

Configure Firewall

Enable port for glusterd service to use GlusterFS Replicated Volume on both the cluster nodes

# firewall-cmd --permanent --add-service=glusterfs
# firewall-cmd --reload

 

Add your nodes to the Trusted Storage Pool (TSP)

Let’s select one host (it doesn’t matter which one); we are going to start our cluster.
We are going to do the following from this one server:

  • Add peers to our cluster
  • Create a GlusterFS replicated volume

To add our peers to the cluster, we issue the following:

[root@glusterfs-1 ~]# gluster peer probe glusterfs-3.example.com
peer probe: success.

[root@glusterfs-1 ~]# gluster peer probe glusterfs-2.example.com
peer probe: success.

To check the connected peer status

[root@glusterfs-1 ~]# gluster peer status
Number of Peers: 2

Hostname: glusterfs-3.example.com
Uuid: 9692eb2e-4655-4922-b0a3-cbbda3aa1a3e
State: Peer in Cluster (Connected)

Hostname: glusterfs-2.example.com
Uuid: 17dd8f27-c595-462b-b62c-71bbebce66ce
State: Peer in Cluster (Connected)

 

Set up GlusterFS Replicated Volume

Below is the syntax used to create glusterfs replicated volume

# gluster volume create NEW-VOLNAME [replica COUNT] [transport [tcp |rdma | tcp,rdma]] NEW-BRICK...

For example here I am creating a new glusterfs replicated volume "replicated_volume" on all my cluster nodes i.e. glusterfs-1, glusterfs-2 and glusterfs-3.

Install & configure glusterfs replicated volume RHEL/CentOS 8

It is going to replicate all the files over the three bricks under the new directory rep_vol which will be created by the below command:

[root@glusterfs-1 ~]# gluster volume create replicated_volume replica 3 glusterfs-1:/bricks/brick1/rep_vol glusterfs-2:/bricks/brick2/rep_vol glusterfs-3:/bricks/brick3/rep_vol
volume create: replicated_volume: success: please start the volume to access data

 

What is Split Brain?

Split brain is where at least two servers serving the same application in a cluster can no longer see each other and yet they still respond to clients. In this situation, data integrity and consistency start to drift apart as both servers continue to serve and store data but can no longer sync any data between each other.

IMPORTANT NOTE:
If you choose to have replica 2 then you may get below WARNING message. Number of bricks must be a multiple of replica count.
Replica 2 volumes are prone to split-brain. Use Arbiter or Replica 3 to avoid this. See: http://docs.gluster.org/en/latest/Administrator%20Guide/Split%20brain%20and%20ways%20to%20deal%20with%20it/.

Next start the volume you created

[root@glusterfs-1 ~]# gluster volume start replicated_volume
volume start: replicated_volume: success

To get more info on the replicated_volume

[root@glusterfs-1 ~]# gluster volume info replicated_volume

Volume Name: replicated_volume
Type: Replicate
Volume ID: a9740dad-0102-4131-8a08-74f3b2ec6103
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: glusterfs-1:/bricks/brick1/rep_vol
Brick2: glusterfs-2:/bricks/brick2/rep_vol
Brick3: glusterfs-3:/bricks/brick3/rep_vol
Options Reconfigured:
features.shard: enable
performance.client-io-threads: off
nfs.disable: on
storage.fips-mode-rchecksum: on
transport.address-family: inet
NOTE:
If the volume does not show "Started", the files under /var/log/glusterfs/glusterd.log should be checked in order to debug and diagnose the situation. These logs can be looked at on one or, all the servers configured.

 

Testing the GlusterFS Replicated Volume

For this step, we will use one of the servers to mount the volume. Typically, you would do this from an external machine, known as a "client". Since using this method would require additional packages to be installed on the client machine, we will use one of the servers as a simple place to test first , as if it were that "client".

On client gluster-fuse rpm must be installed manually

# rpm -Uvh https://buildlogs.centos.org/centos/8/storage/x86_64/gluster-9/Packages/g/glusterfs-fuse-9.4-1.el8.x86_64.rpm

Since I am using one of the gluster nodes, the client package is already installed here

[root@glusterfs-1 ~]# rpm -q glusterfs-fuse
glusterfs-fuse-9.4-1.el8.x86_64

Create a mount point

[root@glusterfs-1 ~]# mkdir /my_repvol

Mount the Gluster Replicated Volume as shown below:

[root@glusterfs-1 ~]# mount -t glusterfs glusterfs-1.example.com:/replicated_volume /my_repvol

[root@glusterfs-1 ~]# df -Th /my_repvol
Filesystem                                 Type            Size  Used Avail Use% Mounted on
glusterfs-1.example.com:/replicated_volume fuse.glusterfs  2.0G   68M  2.0G   4% /my_repvol

Repeat the step on each gluster node

[root@glusterfs-2 ~]# mkdir /my_repvol

[root@glusterfs-2 ~]# mount -t glusterfs glusterfs-1.example.com:/replicated_volume /my_repvol

[root@glusterfs-2 ~]# df -Th /my_repvol
Filesystem                                 Type            Size  Used Avail Use% Mounted on
glusterfs-1.example.com:/replicated_volume fuse.glusterfs  2.0G   68M  2.0G   4% /my_repvol
[root@glusterfs-3 ~]# mkdir /my_repvol

[root@glusterfs-3 ~]# mount -t glusterfs glusterfs-1.example.com:/replicated_volume /my_repvol

[root@glusterfs-3 ~]# df -Th /my_repvol
Filesystem                                 Type            Size  Used Avail Use% Mounted on
glusterfs-1.example.com:/replicated_volume fuse.glusterfs  2.0G   68M  2.0G   4% /my_repvol

Next I will create 5 files on the replicated_volume

[root@glusterfs-1 ~]# touch /my_repvol/file{1..5}

Verify the list of files on the gluster nodes:

[root@glusterfs-1 ~]# ls -l /bricks/brick1/rep_vol/
total 0
-rw-r--r-- 2 root root 0 Jan 27 11:37 file1
-rw-r--r-- 2 root root 0 Jan 27 11:37 file2
-rw-r--r-- 2 root root 0 Jan 27 11:37 file3
-rw-r--r-- 2 root root 0 Jan 27 11:37 file4
-rw-r--r-- 2 root root 0 Jan 27 11:37 file5

[root@glusterfs-2 ~]# ls -l /bricks/brick2/rep_vol/
total 0
-rw-r--r-- 2 root root 0 Jan 27 11:37 file1
-rw-r--r-- 2 root root 0 Jan 27 11:37 file2
-rw-r--r-- 2 root root 0 Jan 27 11:37 file3
-rw-r--r-- 2 root root 0 Jan 27 11:37 file4
-rw-r--r-- 2 root root 0 Jan 27 11:37 file5

[root@glusterfs-3 ~]# ls -l /bricks/brick3/rep_vol/
total 0
-rw-r--r-- 2 root root 0 Jan 27 11:37 file1
-rw-r--r-- 2 root root 0 Jan 27 11:37 file2
-rw-r--r-- 2 root root 0 Jan 27 11:37 file3
-rw-r--r-- 2 root root 0 Jan 27 11:37 file4
-rw-r--r-- 2 root root 0 Jan 27 11:37 file5

So as expected the files are replicated across the gluster nodes.

 

Lastly I hope the steps from the article to install and configure glusterfs replicated volume on RHEL/CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.

Deepak Prasad

Deepak Prasad

Deepak Prasad is the founder of GoLinuxCloud, bringing over a decade of expertise in Linux, Python, Go, Laravel, DevOps, Kubernetes, Git, Shell scripting, OpenShift, Networking, and Security. His extensive experience spans development, DevOps, networking, and security, ensuring robust and efficient solutions for diverse projects.

Certifications and Credentials:

  • Certified Kubernetes Application Developer (CKAD)
  • Go Developer Certification
  • Linux Foundation Certified System Administrator (LFCS)
  • Certified Ethical Hacker (CEH)
  • Python Institute PCAP (Certified Associate in Python Programming)
You can connect with him on his LinkedIn profile and join his Facebook and LinkedIn page.

Can't find what you're searching for? Let us assist you.

Enter your query below, and we'll provide instant results tailored to your needs.

If my articles on GoLinuxCloud has helped you, kindly consider buying me a coffee as a token of appreciation.

Buy GoLinuxCloud a Coffee

For any other feedbacks or questions you can send mail to admin@golinuxcloud.com

Thank You for your support!!

Leave a Comment