In my last article I shared the steps to configure LVM based HA cluster without GFS2 file system. Now let me share the steps to configure iSCSI target and initiator on RHEL/CentOS 7 and 8 Linux node. I am using Virtual Machines running on Oracle VirtualBox installed on my Linux Server
iscsi is an acronym for Internet Small Computer System Interface. We can consider iscsi as a block storage since storage is accessed at the block layer. So basically iSCSI is a block level protocol for sharing RAW storage devices over an IP network. We also call it a SAN technology i.e. iSCSI SAN. Since it operates over IP network, do not mix or confuse it with NAS technologies like NFS or SMB. They also work over IP Network but they operate on File System Layer. but in iSCSI we work on RAW blocks. In this article I will share the steps to configure iscsi target and initiator on RHEL/CentOS 7 and 8.
iSCSI SAN Architecture
When setting up an iSCSI SAN, you configure one server as the iSCSI target. This is the server that offers access to the shared storage devices. When you configure RHEL or CentOS 7 as an iSCSI target, the shared storage devices typically are LVM logical volumes, but they can be complete disks or partitions as well.
The other server is going to be used as the iSCSI initiator. This is the server that connects to the SAN. After connecting to the SAN, the iSCSI initiator sees an additional disk device.
Now iSCSI initiator goes through the process of discovering targets on the network, authenticating and logging in. Eventually accessing these iSCSI LUNs on localhost.
When using a redundant network connection, the iSCSI initiator will see the SAN device twice, once over each different path to the SAN. This might lead to a situation where the same shared disk device is presented twice as well. To make sure that in such a setup where redundant paths are available the SAN device is addressed correctly, the iSCSI initiator should be configured to run the multipath driver.
iSCSI SAN Terminology
| Item | Description |
|---|---|
| IQN | The iSCSI qualified name. A unique name that is used for identifying targets as well as initiators |
| Backend Storage | The storage devices on the iSCSI target that the iSCSI target component is providing access to |
| Target | The service on an iSCSi server that gives access to backend storage devices. |
| Initiator | The iSCSi client that connects to a target and is identified by IQN |
| ACl | The access control list that is based on the iSCSI initiator IQN and used to provide access to specific user |
| LUN | A Logical Unit Number. The backend storage devices that are shared through the target. This can be any device that supports read/write operations, such as disk, partitions, logical volumes, files or tape drves |
| Portal | The IP address and port that a target or initiator uses to establish connections |
| TPG | The Target Portal Group. This is the collection of the IP Address and TCP ports to which a specific iSCSI target will listen. |
| Discovery | The process whereby an initiator finds the targets that are configured on a portal and stores the information locally for future reference. Discovery is done by using the iscsiadm command |
| Login | Authentication that gives an initiator access to LUNs on the target. After successful login, the login information is stored on the initiator automatically. Login is performed using the iscsiadm command |
My Setup Details
| Properties | node1 (Initiator) | storage1(target) |
|---|---|---|
| OS | CentOS 7 | CentOS 7 |
| vCPU | 2 | 2 |
| Memory | 4 GB | 4 GB |
| Disk | 20GB | 20GB |
| Hostname | node1 | storage1 |
| FQDN | node1.example.com | storage1.example.com |
| IP Address | 10.0.2.20 | 10.0.2.13 |
Setting Up the iSCSI Target on RHEL/CentOS 7/8
Throughout different versions of Linux, different iSCSI target packages have been used. In Red Hat Enterprise Linux 7 and 8, the LIO (Linux I/O) target is used. LIO is the standard iSCSI target solution since Linux kernels 2.6.38, it has become an attractive storage solution that has rapidly replaced alternative iSCSI target solutions in many Linux distributions. The default interface to manage the LIO target is the targetcli command. This command uses familiar Linux commands, such as cd, ls, pwd, and set to configure the target.
Steps to setup iSCSI target
- Create the backing storage devices.
- Create the IQN and default target portal group (TPG).
- Configure one or more ACLs for the TPG.
- Create LUNs to provide access to the backing storage devices.
- Create a portal to provide a network interface that iSCSI initiators can connect to.
- Verify and commit the configuration.
1. Create backing storage device
Before we start working on our iSCSI target, we need a backend storage. On my node I have added an additional disk mapped to /dev/sdc. Below using fdisk I am creating a new partition /dev/sdc1 with 1GB size, which will be used to create my iSCSI target.
[root@storage1 ~]# fdisk /dev/sdc Welcome to fdisk (util-linux 2.23.2). Changes will remain in memory only, until you decide to write them. Be careful before using the write command. Command (m for help): n Partition type: p primary (0 primary, 0 extended, 4 free) e extended Select (default p): p Partition number (1-4, default 1): First sector (2048-8388607, default 2048): Using default value 2048 Last sector, +sectors or +size{K,M,G} (2048-8388607, default 8388607): +1G Partition 1 of type Linux and of size 1 GiB is set Command (m for help): p Disk /dev/sdc: 4294 MB, 4294967296 bytes, 8388608 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 512 bytes / 512 bytes Disk label type: dos Disk identifier: 0x243b95e3 Device Boot Start End Blocks Id System /dev/sdc1 2048 2099199 1048576 83 Linux Command (m for help): w The partition table has been altered! Calling ioctl() to re-read partition table. Syncing disks. reload the partition table
Update the partition table
[root@storage1 ~]# partprobe
Validate the new partition
[root@storage1 ~]# ls -l /dev/sd*
brw-rw----. 1 root disk 8, 0 Dec 29 10:13 /dev/sda
brw-rw----. 1 root disk 8, 1 Dec 29 10:13 /dev/sda1
brw-rw----. 1 root disk 8, 2 Dec 29 10:13 /dev/sda2
brw-rw----. 1 root disk 8, 16 Dec 29 10:13 /dev/sdb
brw-rw----. 1 root disk 8, 17 Dec 29 10:13 /dev/sdb1
brw-rw----. 1 root disk 8, 32 Dec 29 10:13 /dev/sdc
brw-rw----. 1 root disk 8, 33 Dec 29 10:13 /dev/sdc1
2. Install targetcli rpm
To manage the kernel-based iSCSI Target service on RHEL/CentOS 7/8, we will need to install the targetcli package, as shown in the following command:
[root@storage1 ~]# yum -y install targetcli
Once successfully installed proceed with the steps to configure iSCSI target on your RHEL or CentOS 7 Linux node.
3. Managing iSCSI targets with targetcli
The targetcli command is a shell to view, edit, save, and load the iSCSI target configuration. When you look at the configuration, you will see that targetcli provides a hierarchical structure in a similar way to a filesystem.
To invoke the targetcli shell, we will run this command as root. You will see that on the first run of the command, a preferences file is created. This is illustrated in the following snippet
[root@storage1 ~]# targetcli targetcli shell version 2.1.fb46 Copyright 2011-2013 by Datera, Inc and others. For help on commands, type 'help'. />
As you can see in the preceding output, you can enter help to display a list of commands that can be entered. To view the available configuration objects, we can use the ls command. The output is shown in the following screenshot:
/> ls o- / ...................................................................................................................... [...] o- backstores ........................................................................................................... [...] | o- block ............................................................................................... [Storage Objects: 0] | o- fileio .............................................................................................. [Storage Objects: 0] | o- pscsi ............................................................................................... [Storage Objects: 0] | o- ramdisk ............................................................................................. [Storage Objects: 0] o- iscsi ......................................................................................................... [Targets: 0]
We will work with backstores objects to start with so that we can add it to the LVM block device in the configuration in addition to the fileio backstore. As the name suggests, this will be a file within the filesystem; we can share this to a network as a virtual disk.
4. Create block backstores
We will work from the root of the targetcli configuration; this should be exactly where we are, but we can always use the pwd command to display our working directory. If required, we can change it to the root of the configuration with cd /.
While using the
targetcli command, we can use CTRL + L to clear the screen as we would in Bash, but most importantly, the Tab key completion works, so we do not need to type the complete name or path to objects and properties.To create a new block, back store on the partition that we created earlier in this section.
[root@storage1 ~]# targetcli
targetcli shell version 2.1.fb46
Copyright 2011-2013 by Datera, Inc and others.
For help on commands, type 'help'.
/backstores/block> create dev=/dev/sdc1 name=sdc1
Created block storage object sdc1 using /dev/sdc1.
This will create the block backstore with a name called sdc1. Using the ls command again will list the additional object within the hierarchy. In the following screenshot, we see the creation of the backstore and the subsequent listing:
/backstores/block> ls
o- block ...................................................................................................... [Storage Objects: 1]
o- sdc1 ............................................................................. [/dev/sdc1 (0 bytes) write-thru deactivated]
o- alua ....................................................................................................... [ALUA Groups: 1]
o- default_tg_pt_gp ........................................................................... [ALUA state: Active/optimized]
To go back to the home directory
/backstores/block> cd /
5. Creating iSCSI targets
The iSCSI objects that we see in the main list represents iSCSI targets and their properties. Firstly, we will create a simple iSCSI target with default names.
/> cd iscsi
Here we will now create an iSCSI target by supplying a custom IQN. To perform this, we create the object and specify the name that is usually written to contain the date and the reversed DNS name. Here we have used a sample IQN
/iscsi> create wwn=iqn.2018-12.com.example:servers
Created target iqn.2018-12.com.example:servers.
Created TPG 1.
Global pref auto_add_default_portal=true
Created default portal listening on all IPs (0.0.0.0), port 3260.
IQN starts with
iqn, which is followed by the year and month it was created and the reverse DNS name. If you specify the month as one digit instead of two, for instance, you’ll get a “WWN not valid” message, and creation will fail.We can add the description of the target with the :servers at the end, indicating that this is a target for the servers.
We can filter what is displayed using the ls command by adding the object hierarchy that we want to list. For example, to list targets, we will use the ls iscsi command.
The output of this command is shown in the following screenshot:
/iscsi> ls
o- iscsi .............................................................................................................. [Targets: 2]
o- iqn.2018-12.com.example:servers ..................................................................................... [TPGs: 1]
o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
o- acls ............................................................................................................ [ACLs: 0]
o- luns ............................................................................................................ [LUNs: 0]
o- portals ...................................................................................................... [Portals: 1]
o- 0.0.0.0:3260 ....................................................................................................... [OK]
Now we have our customized name for the target, but we still have to add the LUNS or logical units to make the SAN (Storage Area Network) effective.
6. Adding ACLs
To create an ACL, we limit the access from LUN to a given initiator name or names that we mention in Access Control List (ACL). The initiator is the iSCSI client and will have a unique client IQN configured on the initiator in the /etc/iscsi/initiatorname.iscsi file.
If this file is not present, you will need to install the
iscsi-initiator-utils package on the initiator node.The filename used to configure the initiator name will be consistent for Linux clients, but will differ for other operating systems. To add an ACL, we will remain with the current configuration hierarchy: /iscsi/iqn….:servers/tpg1 and issue the following command, again written as a single line:
/iscsi> cd iqn.2018-12.com.example:servers/tpg1/acls /iscsi/iqn.20...ers/tpg1/acls> create wwn=iqn.2018-12.com.example:node1 Created Node ACL for iqn.2018-12.com.example:node1 /iscsi/iqn.20...ers/tpg1/acls> cd /
Using the ls command from this location in the configuration hierarchy, we see the output similar to the following screenshot, which also includes the command to create the ACL:
/> ls o- / ......................................................................................................................... [...] o- backstores .............................................................................................................. [...] | o- block .................................................................................................. [Storage Objects: 1] | | o- sdc1 ......................................................................... [/dev/sdc1 (0 bytes) write-thru deactivated] | | o- alua ................................................................................................... [ALUA Groups: 1] | | o- default_tg_pt_gp ....................................................................... [ALUA state: Active/optimized] | o- fileio ................................................................................................. [Storage Objects: 0] | o- pscsi .................................................................................................. [Storage Objects: 0] | o- ramdisk ................................................................................................ [Storage Objects: 0] o- iscsi ............................................................................................................ [Targets: 2] | o- iqn.2018-12.com.example:servers ................................................................................... [TPGs: 1] | o- tpg1 ............................................................................................... [no-gen-acls, no-auth] | o- acls .......................................................................................................... [ACLs: 1] | | o- iqn.2018-12.com.example:node1 ........................................................................ [Mapped LUNs: 0] | o- luns .......................................................................................................... [LUNs: 0] | o- portals .................................................................................................... [Portals: 1] | o- 0.0.0.0:3260 ..................................................................................................... [OK] o- loopback ......................................................................................................... [Targets: 0]
This ACL restricts access to the initiator listed within the ACL. Be careful if you ever change the initiator name because the ACL will also need to be updated. The initiator is the iSCSI client.
7. Adding LUNs to the iSCSI target
Staying with the targetcli shell, we will now move on to our target and TPG (Target Portal Group) object. Similar to the filesystem, this is achieved using the cd command, as shown in the following command:
/> cd iscsi/iqn.2018-12.com.example:servers/tpg1/luns
We have one portal that listens on all IPv4 interfaces on the TCP port 3260. Currently, we have no acls or luns. To add a LUN, we will use the following command, which will utilize the LVM block backstore:
/iscsi/iqn.20...ers/tpg1/luns> create /backstores/block/sdc1
Created LUN 0.
Created LUN 0->0 mapping in node ACL iqn.2018-12.com.example:node1
The iSCSI target is now configured. Once you exit the configuration will be saved to /etc/target/saveconfig.json or you can optionally also run saveconfig on the terminal.
/iscsi/iqn.20...ers/tpg1/luns> exit
Global pref auto_save_on_exit=true
Configuration saved to /etc/target/saveconfig.json
[root@storage1 ~]#
8. Update firewall
Now that the iSCSI target has been configured, you need to make sure that it can be accessed through the firewall and that the service is started automatically.
To open port 3260 in the firewall, execute below commands
[root@storage1 ~]# firewall-cmd --add-port=3260/tcp --permanent [root@storage1 ~]# firewall-cmd --reload
9. Start and enable target service
Now that the iSCSI target has been configured, we need to start and enable the target service
[root@storage1 ~]# systemctl start target [root@storage1 ~]# systemctl enable target
Setting Up the iSCSI Initiator
The iSCSI Initiator or client on RHEL/CentOS 7/8 is installed with the iscsi-initiator-utils package; you can verify that this is installed on your system using the yum command, as shown in the following example:
[root@node1 ~]# rpm -q iscsi-initiator-utils iscsi-initiator-utils-6.2.0.874-7.el7.x86_64
and if not available you can install it using yum
[root@node1 ~]# yum -y install iscsi-initiator-utils
1. Setting the iSCSI Initiatorname
For the purpose of this exercise, we will use a separate RHEL 7 & 8 system as our initiator and connect it to the existing target. We will need to edit the /etc/iscsi/initiatorname.iscsi file on the new RHEL 7 & 8 system to ensure that the name is set to match the name we added to the ACL in the earlier section of this article
[root@node1 ~]# vi /etc/iscsi/initiatorname.iscsi
[root@node1 ~]# cat /etc/iscsi/initiatorname.iscsi
InitiatorName=iqn.2018-12.com.example:node1
So here we have manually updated the file with the ACL name we used on the iSCSI target.
Next restart the iscsid daemon
[root@node1 ~]# systemctl restart iscsid
2. Discover the LUNs
When using iSCSI discovery, you need three different arguments:
- --type sendtargets: This tells the discovery mode how to find the iSCSI targets.
- --portal: This argument tells the
iscsiadmcommand which IP address and port to address to perform the discovery. You can use an IP address or node name as the argument, and optionally, you can specify a port as well. If no port is specified, the default port 3260 is used. - --discover: This argument tells the
iscsidservice to perform a discovery.
We will use the main client tool iscsiadm to discover the iSCSI LUNs on the target.
[root@node1 ~]# iscsiadm --mode discovery --type sendtargets --portal 10.0.2.13 --discover 10.0.2.13:3260,1 iqn.2018-12.com.example:servers
After the discovery below database is updated
[root@node1 ~]# ls -l /var/lib/iscsi/nodes total 8 drw------- 3 root root 4096 Dec 29 19:56 iqn.2018-12.com.example:servers
[root@node1 ~]# ls -l /var/lib/iscsi/send_targets/10.0.2.13,3260/ total 12 lrwxrwxrwx 1 root root 69 Dec 29 19:56 iqn.2018-12.com.example:servers,10.0.2.13,3260,1,default -> /var/lib/iscsi/nodes/iqn.2018-12.com.example:servers/10.0.2.13,3260,1 -rw------- 1 root root 547 Dec 29 19:56 st_config
3. Making the connection
Now, we have seen that we can connect to the iSCSI target and have it sent us the configured LUNS. We should now connect to this LUN and use the same command with the following options:
[root@node1 ~]# iscsiadm --mode node --targetname iqn.2018-12.com.example:servers --login
Logging in to [iface: default, target: iqn.2018-12.com.example:servers, portal: 10.0.2.13,3260] (multiple)
Login to [iface: default, target: iqn.2018-12.com.example:servers, portal: 10.0.2.13,3260] successful.
In this command, a few options are used:
- --mode node: This specifies iscsiadm to enter “
node” mode. This is the mode in which the actual connection with the target can be established. - --targetname: This specifies the name of the target as discovered when using the iSCSI discovery process.
- --portal: This is the IP address and port on which the target is listening.
- --login: This authenticates to the target and will store credentials as well to ensure that on reboot the connection can be reestablished again.
After logging in, a session with the iSCSI target is established. Both the session and the node connection can be monitored, using the -P option
[root@node1 ~]# iscsiadm --mode node -P 1
Target: iqn.2018-12.com.example:servers
Portal: 10.0.2.13:3260,1
Iface Name: default
After making the connection to the iSCSI target, you’ll see the new SCSI devices as offered by the target. A convenient command to list these commands is lsscsi
[root@node1 ~]# lsscsi
[1:0:0:0] cd/dvd VBOX CD-ROM 1.0 /dev/sr0
[2:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sda
[3:0:0:0] disk ATA VBOX HARDDISK 1.0 /dev/sdb
[11:0:0:0] disk LIO-ORG sdc1 4.0 /dev/sdc
4. Managing iSCSI Connection Persistence
After logging in to an iSCSI target server, the connections are persistent automatically. That means that on reboot, the iscsid and iscsi services are started on the iSCSI client, and these services will read the iSCSI configuration that is locally stored to automatically reconnect.
Therefore, there is no need to put anything in configuration files if you have successfully connected once to the iSCSI server.
5. Removing the iSCSI connection
If you need an iSCSI connection not to be restored after reboot, you first have to log out to disconnect the actual session by using below command
[root@node1 ~]# iscsiadm --mode node --targetname iqn.2018-12.com.example:servers --portal 10.0.2.13 -u
Logging out of session [sid: 1, target: iqn.2018-12.com.example:servers, portal: 10.0.2.13,3260]
Logout of [sid: 1, target: iqn.2018-12.com.example:servers, portal: 10.0.2.13,3260] successful.
Next you need to delete the corresponding IQN sub directory and all of its contents. You can do this with the below command
[root@node1 ~]# iscsiadm --mode node --targetname iqn.2018-12.com.example:servers --portal 10.0.2.13 -o delete
Stop the
iscsi.service and remove all files under /var/lib/iscsi/nodes to clean up all current configuration. After doing that, restart the iscsi.service and start the discovery and login again.
6. Mounting iSCSI Devices
To mount an iSCSI device, you need to take care of a few things. First, the iSCSI disk that now appears as /dev/sdc might appear as a different device name the next time it is connected due to a topology change in your SAN configuration. For that reason, it is not a smart idea to put a reference to /dev/sdc in the /etc/fstab file. You should instead use a file system UUID. Every file system automatically gets a UUID.
To request the value of that UUID, you can use the blkid command
[root@node1 ~]# blkid /dev/sdc
/dev/sdc: UUID="f87DLO-DXDO-jjJ5-3vgO-RfCE-oOCA-VGploa" TYPE="LVM2_member"
The second issue when making persistent iSCSI mounts is that normally the
/etc/fstab file is processed before the network is available. To make sure the iSCSI disk can be mounted, you need to use the _netdev mount option in /etc/fstab.
So to ensure that an iSCSI mount is configured persistently, put an entry in /etc/fstab that looks like this:
UUID-XXXXXXXX-XXXX-XXXX-XXXXXXXX /iscsi ext4 _netdev 0 2
Lastly I hope the steps from the article to configure iSCSI target and iSCSI initiator on RHEL / CentOS 7 and 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.
References:
Creating iSCSI target and Initiator in RHEL 7
Creating iscsi traget and initiator in RHEL 8
