Set up KVM PXE server to perform network boot RHEL CentOS 8

In this article I will share the steps to setup KVM PXE server which you can use to perform network based installation of Virtual Machines.

You can use traditional PXE server for performing network based installation. But the steps to setup KVM PXE server for qemu netwoot are different.

We will not use separate DHCP server to install KVM over network, instead we will use default NAT network which will act as DHCP server for QEMU netboot.

 

Lab Environment

I have a physical Linux server installed with RHEL 8.1 with below network connections

[root@rhel-8 ~]# nmcli con show
NAME    UUID                                  TYPE      DEVICE
br0     4a38747e-49f4-4213-83cb-a946d3dffefb  bridge    nm-bridge
virbr0  63200e70-21fb-4e89-b9f2-031d9eb71ab3  bridge    virbr0
slave1  eb862842-ba6a-47fe-994c-733977a8275b  ethernet  eno49
slave2  631e5c3e-bf2f-46d3-b9dd-dca2001f586b  ethernet  eno50
vnet0   3960b1c1-6453-40af-8d3f-aa0d15eeb0f5  tun       vnet0
vnet1   409837ab-9814-4041-8071-8631ba323cce  tun       vnet1

I have manually created br0 bridge connection using nmcli for my KVM Virtual Machines. When you install KVM and Virtualization related rpms, by default in RHEL/CentOS, a NAT network with bridge connection virbr0 is created on the Linux server.

To list the IP Address details of virbr0 and my custom nm-bridge interface

[root@rhel-8 ~]# ip address show dev virbr0
98: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:5d:42:fc brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
	   
[root@rhel-8 ~]# ip address show dev nm-bridge
85: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 00:17:a4:77:00:44 brd ff:ff:ff:ff:ff:ff
    inet 10.43.138.12/27 brd 10.43.138.31 scope global noprefixroute nm-bridge
       valid_lft forever preferred_lft forever

This is the default network we talked about earlier, which is the NAT network created once we install KVM and other Virtualization rpms

[root@rhel-8 ~]# virsh net-list
 Name                 State      Autostart     Persistent
----------------------------------------------------------
 default              active     yes           yes

 

Step 1: Setup Installation Repo

We must copy the content of our RHEL/CentOS ISO DVD to some directory to set up installation repo We will keep the content of our CentOS 8 DVD under /images/

[root@rhel-8 ~]# mkdir /images

I have virtually mounted RHEL 8 ISO to my virtual machine, to access this ISO I will mount the image to /mnt

[root@rhel-8 ~]# mount /dev/sr0 /mnt
mount: /mnt: WARNING: device write-protected, mounted read-only.

Copy all the files from the CentOS 8 ISO Image to /images directory

[root@rhel-8 ~]# cp -apr /mnt/* /images/

Note that here copying of hidden file will be ignored and .treeinfo file which is required for a valid installation source so manually copy these hidden files.

[root@rhel-8 ~]# cp -apr /mnt/.discinfo /mnt/.treeinfo /images/

Verify the repository content

[root@rhel-8 ~]# ls -al /images
dr-xr-xr-x.  7 root root 4096 May  4 20:09 .
dr-xr-xr-x. 22 root root 4096 May  1 03:30 ..
dr-xr-xr-x.  4 root root 4096 Jan  4 03:12 AppStream
dr-xr-xr-x.  4 root root 4096 Jan  4 03:12 BaseOS
-r--r--r--.  1 root root   44 Jan  4 03:11 .discinfo
dr-xr-xr-x.  3 root root 4096 Jan  4 03:12 EFI
dr-xr-xr-x.  3 root root 4096 Jan  4 03:12 images
dr-xr-xr-x.  2 root root 4096 Jan  4 03:12 isolinux
-r--r--r--.  1 root root   87 Jan  4 03:11 media.repo
-r--r--r--.  1 root root  664 Jan  4 03:12 TRANS.TBL
-r--r--r--.  1 root root 1520 Apr 30 00:08 .treeinfo

 

Step 2: Install and Configure TFTP Server

We would need TFTP server to share installation files on our KVM PXE server such as initrd, vmlinuz and other related files.

Install tftp-serverusing dnf on your KVM Host. I hope you are aware of YUM's alternative, DNF tool in RHEL/CentOS 8

NOTE:

On RHEL system you must have an active subscription to RHN or you can configure a local offline repository using which "yum" package manager can install the provided rpm and it's dependencies.
[root@rhel-8 ~]# dnf install tftp-server -y

Earlier with RHEL/CentOS 7, tftp service was managed by xinetd. Although even with RHEL/CentOS 7 we had an option to disable xinetd and fork TFTP process using systemd.

NOTE:

You can fork TFTP process either by xinetd or by systemd but not both. For more details check: TFTP service is getting inactive automatically in RHEL7

In this example we will fork TFTP process using systemd.
The tftp-server rpm installation will create /usr/lib/systemd/system/tftp.service unit file

Next start the tftp service and enable it to start automatically post reboot

[root@rhel-8 ~]# systemctl enable tftp.socket --now
  • TFTP service uses to tftp.socket to serve TFTP requests
  • So it is possible that if there are no incoming TFTP requests then the tftp service will become inactive on it's own
  • But as soon as a TFTP request goes to tftp.socket, the socket will start tftp.service and serve the request
  • Check the status of tftp.socket
[root@rhel-8 ~]# systemctl status tftp.socket
● tftp.socket - Tftp Server Activation Socket
   Loaded: loaded (/usr/lib/systemd/system/tftp.socket; enabled; vendor preset: disabled)
   Active: active (listening) since Wed 2020-04-29 21:44:24 IST; 5h 41min ago
   Listen: [::]:69 (Datagram)
   CGroup: /system.slice/tftp.socket

Apr 29 21:44:24 rhel-8.example.com systemd[1]: Listening on Tftp Server Activation Socket.

To perform qemu netboot we will configure our NAT network to provide IP and TFTP directory location.

 

Step 3: Setup "default" NAT network as DHCP Server

To edit the "default" network use virsh command

[root@rhel-8 ~]# virsh net-edit default
<network>
  <name>default</name>
  <uuid>836520d9-fb0a-46f0-8745-033353151e93</uuid>
  <forward mode='nat'/>
  <bridge name='virbr0' stp='on' delay='0'/>
  <mac address='52:54:00:5d:42:fc'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <tftp root='/var/lib/tftpboot/pxelinux'/>
    <dhcp>
      <range start='192.168.122.2' end='192.168.122.254'/>
      <bootp file='pxelinux.0'/>
    </dhcp>
  </ip>
</network>

Here as highlighted I have added TFTP directory location and pxelinux.0 file required to perform qemu netboot on our KVM PXE server.
Press Esc+wq! to save and exit this file.

To refresh the changes, first we need to destroy the default network. Don't worry this will not delete the default network.

[root@rhel-8 ~]# virsh net-destroy default
Network default destroyed

Next start the default network

[root@rhel-8 ~]# virsh net-start default
Network default started

 

Step 4: Setup KVM PXE Server

  • To perform QEMU netboot on our KVM Virtual Virtual Machines, we must setup KVM PXE server on the host where KVM was installed with required installation files
  • We will need Linux boot images to boot the RHEL 8 OS with minimal configuration
  • This is performed using initrd and vmlinuz. I hope you are familiar with the Linux boot process which covers this part
  • Before the actual kernel loads, initrd and vmlinuz will load the necessary drivers from the memory to boot up the server

Step 4.1: Extract syslinux-tftpboot

With RHEL/CentOS 8, the pxelinux file is part of syslinux-tftpboot rpm so we will copy this file from our CentOS 8 ISO to a temporary location

[root@rhel-8 ~]# cp /mnt/BaseOS/Packages/syslinux-tftpboot-6.04-4.el8.noarch.rpm /tmp/

Next extract the syslinux-tftpboot rpm. We will not use all the contents of this rpm and only required files

[root@rhel-8 ~]# cd /tmp/
[root@rhel-8 tmp]# rpm2cpio syslinux-tftpboot-6.04-4.el8.noarch.rpm | cpio -idm

The above command will extract syslinux-tftpboot under /tmp.

Next we will copy pxelinux.0 and ldlinux.c32 to /var/lib/tftpboot/pxelinux/ required to setup PXE boot server

[root@rhel-8 tmp]# cp /tmp/tftpboot/ldlinux.c32 /var/lib/tftpboot/pxelinux/
[root@rhel-8 tmp]# cp /tmp/tftpboot/pxelinux.0 /var/lib/tftpboot/pxelinux/

 

Step 4.2: Copy initrd and vmlinuz

  • We also need other PXE boot images which will be under isolinux folder of the RHEL/CentOS 8 Image.
  • Since we had copied the ISO content to /images, we will copy required PXE boot images from /images/isolinux to /var/lib/tftpboot/pxelinux/
[root@rhel-8 tmp]# cp /images/isolinux/initrd.img /var/lib/tftpboot/pxelinux/
[root@rhel-8 tmp]# cp /images/isolinux/vmlinuz /var/lib/tftpboot/pxelinux/

List all the PXE boot image files:

[root@rhel-8 tmp]# ls -l /var/lib/tftpboot/pxelinux/
total 58880
-r--r--r-- 1 root root 62248424 Apr 19 19:14 initrd.img
-rw-r--r-- 1 root root   116096 Apr 19 19:47 ldlinux.c32
-rw-r--r-- 1 root root    42821 Apr 19 19:01 pxelinux.0
-r-xr-xr-x 1 root root  8106848 Apr 19 19:14 vmlinuz

Next navigate to pxelinux folder

[root@rhel-8 tmp]# cd /var/lib/tftpboot/pxelinux

 

Step 4.3: Create Boot Menu

  • We will create KVM PXE boot server which can be used to install multiple images so we will create a boot menu.
  • This boot menu will be displayed to initiate the installation for user input.
  • Create a new file boot.msg under /var/lib/tftpboot/pxelinux
  • You can use any name for this file as per your requirement.
  • Since I intent to configure Kickstart server only for single Image, I have only created two menu entries.
[root@rhel-8 pxelinux]# cat boot.msg
Welcome to the installation of "My Linux Server" !
Red Hat Enterprise linux 8.1 (x86_64)
Version: 1.0
Architecture: x86_64

To start the installation enter :
    '1', '2' and press .

Available boot options:

  1 - Install Red Hat Enterprise Linux 8.1
  2 - Boot from Harddisk (this is default)

Have a lot of fun...

 

Step 5: Create PXE configuration file

  • Once the client retrieves and executes pxelinux.0, it is hard-coded to look for a file from the pxelinux.cfg/ sub directory relative to where pxelinux.0 was found
  • In large deployments we create individual PXE configuration file per node.
  • The naming syntax of these PXE configuration file is very important
  • First, it will look for a file named after the MAC address, in the form 01-xx-xx-xx-xx-xx-xx
  • For example, if NIC's MAC Address is 08:00:27:83:1e:2a so the PXE configuration file will be 01-08-00-27-83-1e-2a
  • Next, it will look for a file named by the IP address as provided by the DHCP server.
  • The IP address is looked up in hexadecimal format.
  • You can use printf to get the hexadecimal format of an IP Address, for example to get the hexadecimal value of 10.10.10.12
# printf "%02x%02x%02x%02xn" 10 10 10 12
0a0a0a0cn
  • I have created multiple PXE configuration file for our KVM Virtual Machines
  • We don't need to worry about MAC as we will assign custom MAC when we create KVM Virtual Machine
[root@rhel-8 pxelinux.cfg]# ls -l
total 20
-rw-r--r--. 1 root root 320 Apr 30 15:32 01-00-17-a4-77-00-45
-rw-r--r--. 1 root root 320 Apr 30 15:38 01-00-17-a4-77-00-46
-rw-r--r--. 1 root root 320 Apr 30 15:39 01-00-17-a4-77-00-47
-rw-r--r--. 1 root root 320 May  4 20:02 01-00-17-a4-77-00-48

Below is the content of one of my PXE configuration file

[root@rhel-8 pxelinux.cfg]# cat 01-00-17-a4-77-00-45
timeout 600
display boot.msg
default 1
prompt  1

label 1
  menu label ^Install Red Hat Enterprise Linux 8
  kernel vmlinuz
  append initrd=initrd.img showopts  ks=nfs:192.168.122.1://ks/kickstart_centos8-2.conf ip=dhcp net.ifnames=0 biosdevname=0

label 2
  menu label Boot from ^local drive
  localboot 0x80

menu end
  • You must modify the kernel arguments based on your kickstart file name and path
  • label 2 would be the default option to boot from hard disk
  • A DHCP IP will be collected from the "default" NAT network
  • net.ifnames and biosdevname are also optional, as I do not wish to use consistent network device naming scheme for my client node.

 

Step 6: Configure Kickstart server

  • With every Red Hat and CentOS installation, a default kickstart file is created under home folder of root user i.e. /root/anaconda-ks.cfg
  • This anaconda kickstart file contains the values used to install your server
  • We can use this anaconda file to create our kickstart configuration file and configure kickstart server
  • You can also use online kickstart generator tool provided by Red Hat.

Create /ks directory where we will store our kickstart file

[root@rhel-8 pxelinux.cfg]# mkdir /ks

I have created multiple kickstart configuration files for different KVM Virtual Machines on our KVM PXE server

[root@rhel-8 pxelinux.cfg]# cd /ks/
[root@rhel-8 ks]# ls -l
total 20
-rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-2.conf
-rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-3.conf
-rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-4.conf
-rw-r--r--. 1 root root 1352 May  4 20:03 kickstart_centos8-5.conf

 

Step 6.1: Sample kickstart configuration file

Below is my sample kickstart configuration file on the KVM PXE server. I prefer to perform text based network installation and later you can choose to install Graphical packages for GNOME Desktop to your Linux server.

I won't be able to explain the entire content of kickstart file in this article as it would be too long, I have written another article with detailed description of all the values from kickstart file.

#version=RHEL8

# Use text install
text

# Create new user
user --name=deepak --shell=/bin/bash --homedir=/home/deepak  --iscrypted --password=$6$uSejt/TeWMJVQ/F8$/oFsIanDHS/5b9ssy7gZbQwNkORgRjsQIw4JyFjlTWDh9TVsEXWEy2APpCUNTHipOOEe..ubg3qBZOwpaPtsB.

# Create new repo
repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream

# Use NFS server for installation
nfs --server=192.168.122.1 --dir=/images/

# Keyboard layouts
keyboard --vckeymap=us --xlayouts='us'

# System language
lang en_US.UTF-8

# Network information
network  --bootproto=static --ip=192.168.122.10 --netmask=255.255.255.0 --gateway=192.168.122.1 --device=eth0 --noipv6 --no-activate
network  --hostname=centos8-2.example.com

# Root password
rootpw --iscrypted $6$w7El/FYx9mbTG6x9$Te.Yg6dq0TsQwGpdSjeDGSw4J9ZBAkLXzT9ODMV7I7lHvX3n5.9PCS4jIkS2GbVLZOpVRLvrua3wwbwA.cfWX.

# Run the Setup Agent on first boot
firstboot --enable

# Do not configure the X Window System
skipx

# System services
services --enabled="chronyd"

# System timezone
timezone Asia/Kolkata --isUtc

# Reboot the node post installation
reboot

# Partition clearing information
clearpart --all

# Disk partitioning information
autopart --type=LVM --fstype=ext4

# List of packages to be installed
%packages
@Virtualization Host
kexec-tools

%end

# Add on
%addon com_redhat_kdump --disable

%end

 

Step 7: Install and Configure NFS

I have already written a separate article with detailed description and steps to install and configure both NFSv4 and NFSv3 server. Hence I will be very brief here:

Install nfs-utils rpm required to configure NFS on your KVM PXE server

[root@rhel-8 ~]# dnf -y install nfs-utils
  • Below are the directories I plan to share for my Linux kickstart server.
  • Here /ks contains the kickstart configuration file
  • and /images contains the RHEL 8 ISO content for installation
[root@rhel-8 ~]# cat /etc/exports
/ks     *(ro,sync,no_root_squash)
/images *(ro,sync,no_root_squash)

Re-export the shares

[root@rhel-8 ~]# exportfs -r

Print the available shares

[root@rhel-8 ~]# exportfs -v
/ks             (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,no_root_squash,no_all_squash)
/images         (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,no_root_squash,no_all_squash)

Enable and start the nfs-server.service. Here since we are using NFSV4, we have not started or installed rpcbind.
You can read more at: Step-by-Step Guide to install and configure NFSv4 and NFSv3 server & client in RHEL/CentOS 7/8

[root@rhel-8 ~]# systemctl enable nfs-server --now

Make sure nfs-server service up and running

[root@rhel-8 ~]# systemctl is-active nfs-server
active

 

Step 8: Create KVM VM and perform qemu netboot

  • We will use virt-install tool to create KVM Virtual Machine
  • Using --network and mac.address we will assign a custom MAC address to our primary NIC interface to perform qemu netboot of the VM
  • The MAC address is mapped to our PXE configuration files
  • Use --pxe to perform network based installation
[root@rhel-8 ~]# virt-install --name centos8-2 --memory 10240 --vcpus=2 --network bridge:virbr0,mac.address=00:17:a4:77:00:45  --pxe  --graphics=vnc -v  --disk path=/disks/centos8-2.qcow2,size=20 

Starting install...
Domain installation still in progress.
Waiting for installation to complete.
Domain has shutdown. Continuing.
Domain creation completed.
Restarting guest.

Once the above command executes, you can check your system logs using journalctl, as you see we have DHCP messages to release IP, followed by TFTP messages to transfer boot images

Apr 30 15:34:31 rhel-8.example.com dnsmasq-dhcp[664]: DHCPDISCOVER(virbr0) 00:17:a4:77:00:45
Apr 30 15:34:31 rhel-8.example.com dnsmasq-dhcp[664]: DHCPOFFER(virbr0) 192.168.122.170 00:17:a4:77:00:45
Apr 30 15:34:35 rhel-8.example.com dnsmasq-dhcp[664]: DHCPDISCOVER(virbr0) 00:17:a4:77:00:45
Apr 30 15:34:35 rhel-8.example.com dnsmasq-dhcp[664]: DHCPOFFER(virbr0) 192.168.122.170 00:17:a4:77:00:45
Apr 30 15:34:43 rhel-8.example.com dnsmasq-dhcp[664]: DHCPREQUEST(virbr0) 192.168.122.170 00:17:a4:77:00:45
Apr 30 15:34:43 rhel-8.example.com dnsmasq-dhcp[664]: DHCPACK(virbr0) 192.168.122.170 00:17:a4:77:00:45
Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/pxelinux.0 to 192.168.122.170
Apr 30 15:34:47 rhel-8.example.com dnsmasq-script[664]: Unsupported action: tftp
Apr 30 15:34:47 rhel-8.example.com dnsmasq[664]: script process exited with status 1
Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/ldlinux.c32 to 192.168.122.170
Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/pxelinux.cfg/01-00-17-a4-77-00-45 to 192.168.122.170
Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/boot.msg to 192.168.122.170

 

So our KVM PXE server is running as expected.

Now you don't have to go through so many steps to create KVM Virtual machine every time and your one click install server is ready.

 

Lastly I hope the steps from the article to configure KVM PXE server on RHEL/CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.

2 thoughts on “Set up KVM PXE server to perform network boot RHEL CentOS 8”

  1. Can you reply if its possible to make sth similar like you did but instead using virbr0 as DHCP, make it on bridged network with dedicated dhcp server? I mean I put DHCP, TFTP and NFS server on one VM RHEL 8(VMware) and want to have it as PXE Server and then install from it (nested vm using kvm). However when booting I got message:

    Nothing to boot: No such file or directory (http://ipxe.org/2d03e13b)
    No more network devices

    Address from DHCP is correctly given. Also added to bridge configuration file under libvirt and which looks like the below:

    My DHCP conf:

    allow booting;
    max-lease-time 1200;
    default-lease-time 900;
    log-facility local7;

    option space pxelinux;
    option pxelinux.magic code 208 = string;
    option pxelinux.configfile code 209 = text;
    option pxelinux.pathprefix code 210 = text;
    option pxelinux.reboottime code 211 = unsigned integer 32;
    option architecture-type code 93 = unsigned integer 16;

    subnet 192.168.4.0 netmask 255.255.255.0 {
    option routers 192.168.4.2;
    option domain-name-servers 127.0.0.1;
    range 192.168.4.210 192.168.4.249;

    class "pxeclients" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
    next-server server1.example.com;
    filename "pxelinux/pxelinux.0";
    }
    }

    My /var/lib/tftpboot/pxelinux/pxelinux.cfg/01-08-00-27-99-A7-5D:

    imeout 600
    display boot.msg
    default 1
    prompt 1

    label 1
    menu label ^Install Red Hat Enterprise Linux 8
    kernel vmlinuz
    append initrd=initrd.img showopts ks=nfs:192.168.4.200//ks/ks.cfg ip=dhcp
    label 2
    menu label Boot from ^local drive
    localboot 0x80

    Image from virt-manager:

    https://imgur.com/0jtVmha

    Reply
    • Hello,
      Do you have any logs from the server side during the PXE stage?
      Looks like the DHCP stage has passed and the client got IP 192.168.4.145
      So the problem is with TFTP. Few things you can try
      1. Disable firewall
      # systemctl stop firewalld

      2. I see you are using next-server as server1.example.com. I would recommend putting IP address instead unless you have an internal DNS configured which can resolve the hostname
      3. Install tftp on the server and verify if your tftp-server is working
      tftp localhost
      get vmlinuz

      Reply

Leave a Comment

Please use shortcodes <pre class=comments>your code</pre> for syntax highlighting when adding code.