In this article I will share the steps to setup KVM PXE server which you can use to perform network based installation of Virtual Machines.
You can use traditional PXE server for performing network based installation. But the steps to setup KVM PXE server for qemu netwoot are different.
We will not use separate DHCP server to install KVM over network, instead we will use default NAT network which will act as DHCP server for QEMU netboot.
Lab Environment
I have a physical Linux server installed with RHEL 8.1 with below network connections
[root@rhel-8 ~]# nmcli con show
NAME UUID TYPE DEVICE
br0 4a38747e-49f4-4213-83cb-a946d3dffefb bridge nm-bridge
virbr0 63200e70-21fb-4e89-b9f2-031d9eb71ab3 bridge virbr0
slave1 eb862842-ba6a-47fe-994c-733977a8275b ethernet eno49
slave2 631e5c3e-bf2f-46d3-b9dd-dca2001f586b ethernet eno50
vnet0 3960b1c1-6453-40af-8d3f-aa0d15eeb0f5 tun vnet0
vnet1 409837ab-9814-4041-8071-8631ba323cce tun vnet1
I have manually created br0
bridge connection using nmcli
for my KVM Virtual Machines. When you install KVM and Virtualization related rpms, by default in RHEL/CentOS, a NAT network with bridge connection virbr0
is created on the Linux server.
To list the IP Address details of virbr0
and my custom nm-bridge
interface
[root@rhel-8 ~]# ip address show dev virbr0 98: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 52:54:00:5d:42:fc brd ff:ff:ff:ff:ff:ff inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0 valid_lft forever preferred_lft forever [root@rhel-8 ~]# ip address show dev nm-bridge 85: nm-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000 link/ether 00:17:a4:77:00:44 brd ff:ff:ff:ff:ff:ff inet 10.43.138.12/27 brd 10.43.138.31 scope global noprefixroute nm-bridge valid_lft forever preferred_lft forever
This is the default
network we talked about earlier, which is the NAT network created once we install KVM and other Virtualization rpms
[root@rhel-8 ~]# virsh net-list
Name State Autostart Persistent
----------------------------------------------------------
default active yes yes
Step 1: Setup Installation Repo
We must copy the content of our RHEL/CentOS ISO DVD to some directory to set up installation repo We will keep the content of our CentOS 8 DVD under /images/
[root@rhel-8 ~]# mkdir /images
I have virtually mounted RHEL 8 ISO to my virtual machine, to access this ISO I will mount the image to /mnt
Do you know how to identify and access USB drives, ISO Images connected to a Linux server?
[root@rhel-8 ~]# mount /dev/sr0 /mnt mount: /mnt: WARNING: device write-protected, mounted read-only.
Copy all the files from the CentOS 8 ISO Image to /images
directory
[root@rhel-8 ~]# cp -apr /mnt/* /images/
Note that here copying of hidden file will be ignored and .treeinfo
file which is required for a valid installation source so manually copy these hidden files.
~8GB
of free space to copy the complete RHEL 8 ISO image[root@rhel-8 ~]# cp -apr /mnt/.discinfo /mnt/.treeinfo /images/
Verify the repository content
[root@rhel-8 ~]# ls -al /images dr-xr-xr-x. 7 root root 4096 May 4 20:09 . dr-xr-xr-x. 22 root root 4096 May 1 03:30 .. dr-xr-xr-x. 4 root root 4096 Jan 4 03:12 AppStream dr-xr-xr-x. 4 root root 4096 Jan 4 03:12 BaseOS -r--r--r--. 1 root root 44 Jan 4 03:11 .discinfo dr-xr-xr-x. 3 root root 4096 Jan 4 03:12 EFI dr-xr-xr-x. 3 root root 4096 Jan 4 03:12 images dr-xr-xr-x. 2 root root 4096 Jan 4 03:12 isolinux -r--r--r--. 1 root root 87 Jan 4 03:11 media.repo -r--r--r--. 1 root root 664 Jan 4 03:12 TRANS.TBL -r--r--r--. 1 root root 1520 Apr 30 00:08 .treeinfo
Step 2: Install and Configure TFTP Server
We would need TFTP server to share installation files on our KVM PXE server such as initrd
, vmlinuz
and other related files.
Install tftp-server
using dnf
on your KVM Host. I hope you are aware of YUM's alternative, DNF tool in RHEL/CentOS 8
[root@rhel-8 ~]# dnf install tftp-server -y
Earlier with RHEL/CentOS 7, tftp
service was managed by xinetd
. Although even with RHEL/CentOS 7 we had an option to disable xinetd
and fork TFTP process using systemd
.
In this example we will fork TFTP process using systemd
.
The tftp-server
rpm installation will create /usr/lib/systemd/system/tftp.service
unit file
Next start the tftp
service and enable it to start automatically post reboot
[root@rhel-8 ~]# systemctl enable tftp.socket --now
- TFTP service uses to
tftp.socket
to serve TFTP requests - So it is possible that if there are no incoming TFTP requests then the
tftp
service will become inactive on it's own - But as soon as a TFTP request goes to
tftp.socket
, the socket will starttftp.service
and serve the request - Check the status of
tftp.socket
[root@rhel-8 ~]# systemctl status tftp.socket
● tftp.socket - Tftp Server Activation Socket
Loaded: loaded (/usr/lib/systemd/system/tftp.socket; enabled; vendor preset: disabled)
Active: active (listening) since Wed 2020-04-29 21:44:24 IST; 5h 41min ago
Listen: [::]:69 (Datagram)
CGroup: /system.slice/tftp.socket
Apr 29 21:44:24 rhel-8.example.com systemd[1]: Listening on Tftp Server Activation Socket.
To perform qemu netboot we will configure our NAT network to provide IP and TFTP directory location.
Step 3: Setup "default" NAT network as DHCP Server
To edit the "default
" network use virsh
command
virsh - command not found
then you must first install virsh
tool by installing libvirt-client
on RHEL/CentOS 8 using yum -y install libvirt-client
[root@rhel-8 ~]# virsh net-edit default <network> <name>default</name> <uuid>836520d9-fb0a-46f0-8745-033353151e93</uuid> <forward mode='nat'/> <bridge name='virbr0' stp='on' delay='0'/> <mac address='52:54:00:5d:42:fc'/> <ip address='192.168.122.1' netmask='255.255.255.0'> <tftp root='/var/lib/tftpboot/pxelinux'/> <dhcp> <range start='192.168.122.2' end='192.168.122.254'/> <bootp file='pxelinux.0'/> </dhcp> </ip> </network>
Here as highlighted I have added TFTP directory location and pxelinux.0
file required to perform qemu netboot on our KVM PXE server.
Press Esc+wq!
to save and exit this file.
To refresh the changes, first we need to destroy the default network. Don't worry this will not delete the default network.
[root@rhel-8 ~]# virsh net-destroy default
Network default destroyed
Next start the default network
[root@rhel-8 ~]# virsh net-start default
Network default started
Step 4: Setup KVM PXE Server
- To perform QEMU netboot on our KVM Virtual Virtual Machines, we must setup KVM PXE server on the host where KVM was installed with required installation files
- We will need Linux boot images to boot the RHEL 8 OS with minimal configuration
- This is performed using
initrd
andvmlinuz
. I hope you are familiar with the Linux boot process which covers this part - Before the actual kernel loads,
initrd
andvmlinuz
will load the necessary drivers from the memory to boot up the server
Step 4.1: Extract syslinux-tftpboot
With RHEL/CentOS 8, the pxelinux
file is part of syslinux-tftpboot
rpm so we will copy this file from our CentOS 8 ISO to a temporary location
[root@rhel-8 ~]# cp /mnt/BaseOS/Packages/syslinux-tftpboot-6.04-4.el8.noarch.rpm /tmp/
Next extract the syslinux-tftpboot
rpm. We will not use all the contents of this rpm and only required files
[root@rhel-8 ~]# cd /tmp/ [root@rhel-8 tmp]# rpm2cpio syslinux-tftpboot-6.04-4.el8.noarch.rpm | cpio -idm
The above command will extract syslinux-tftpboot
under /tmp
.
Next we will copy pxelinux.0 and ldlinux.c32 to /var/lib/tftpboot/pxelinux/ required to setup PXE boot server
/var/lib/tftpboot/pxelinux/
[root@rhel-8 tmp]# cp /tmp/tftpboot/ldlinux.c32 /var/lib/tftpboot/pxelinux/ [root@rhel-8 tmp]# cp /tmp/tftpboot/pxelinux.0 /var/lib/tftpboot/pxelinux/
Step 4.2: Copy initrd and vmlinuz
- We also need other PXE boot images which will be under
isolinux
folder of the RHEL/CentOS 8 Image. - Since we had copied the ISO content to
/images
, we will copy required PXE boot images from/images/isolinux
to/var/lib/tftpboot/pxelinux/
[root@rhel-8 tmp]# cp /images/isolinux/initrd.img /var/lib/tftpboot/pxelinux/ [root@rhel-8 tmp]# cp /images/isolinux/vmlinuz /var/lib/tftpboot/pxelinux/
List all the PXE boot image files:
[root@rhel-8 tmp]# ls -l /var/lib/tftpboot/pxelinux/ total 58880 -r--r--r-- 1 root root 62248424 Apr 19 19:14 initrd.img -rw-r--r-- 1 root root 116096 Apr 19 19:47 ldlinux.c32 -rw-r--r-- 1 root root 42821 Apr 19 19:01 pxelinux.0 -r-xr-xr-x 1 root root 8106848 Apr 19 19:14 vmlinuz
Next navigate to pxelinux
folder
[root@rhel-8 tmp]# cd /var/lib/tftpboot/pxelinux
Step 4.3: Create Boot Menu
- We will create KVM PXE boot server which can be used to install multiple images so we will create a boot menu.
- This boot menu will be displayed to initiate the installation for user input.
- Create a new file
boot.msg
under/var/lib/tftpboot/pxelinux
- You can use any name for this file as per your requirement.
- Since I intent to configure Kickstart server only for single Image, I have only created two menu entries.
[root@rhel-8 pxelinux]# cat boot.msg Welcome to the installation of "My Linux Server" ! Red Hat Enterprise linux 8.1 (x86_64) Version: 1.0 Architecture: x86_64 To start the installation enter : '1', '2' and press . Available boot options: 1 - Install Red Hat Enterprise Linux 8.1 2 - Boot from Harddisk (this is default) Have a lot of fun...
Step 5: Create PXE configuration file
- Once the client retrieves and executes
pxelinux.0
, it is hard-coded to look for a file from thepxelinux.cfg/
sub directory relative to wherepxelinux.0
was found - In large deployments we create individual PXE configuration file per node.
- The naming syntax of these PXE configuration file is very important
- First, it will look for a file named after the MAC address, in the form
01-xx-xx-xx-xx-xx-xx
- For example, if NIC's MAC Address is
08:00:27:83:1e:2a
so the PXE configuration file will be01-08-00-27-83-1e-2a
- Next, it will look for a file named by the IP address as provided by the DHCP server.
- The IP address is looked up in hexadecimal format.
- You can use
printf
to get the hexadecimal format of an IP Address, for example to get the hexadecimal value of10.10.10.12
# printf "%02x%02x%02x%02xn" 10 10 10 12
0a0a0a0cn
- I have created multiple PXE configuration file for our KVM Virtual Machines
- We don't need to worry about MAC as we will assign custom MAC when we create KVM Virtual Machine
[root@rhel-8 pxelinux.cfg]# ls -l total 20 -rw-r--r--. 1 root root 320 Apr 30 15:32 01-00-17-a4-77-00-45 -rw-r--r--. 1 root root 320 Apr 30 15:38 01-00-17-a4-77-00-46 -rw-r--r--. 1 root root 320 Apr 30 15:39 01-00-17-a4-77-00-47 -rw-r--r--. 1 root root 320 May 4 20:02 01-00-17-a4-77-00-48
Below is the content of one of my PXE configuration file
[root@rhel-8 pxelinux.cfg]# cat 01-00-17-a4-77-00-45
timeout 600
display boot.msg
default 1
prompt 1
label 1
menu label ^Install Red Hat Enterprise Linux 8
kernel vmlinuz
append initrd=initrd.img showopts ks=nfs:192.168.122.1://ks/kickstart_centos8-2.conf ip=dhcp net.ifnames=0 biosdevname=0
label 2
menu label Boot from ^local drive
localboot 0x80
menu end
- You must modify the kernel arguments based on your kickstart file name and path
label 2
would be thedefault
option to boot from hard disk- A DHCP IP will be collected from the "
default
" NAT network net.ifnames
andbiosdevname
are also optional, as I do not wish to use consistent network device naming scheme for my client node.
Step 6: Configure Kickstart server
- With every Red Hat and CentOS installation, a default kickstart file is created under home folder of root user i.e.
/root/anaconda-ks.cfg
- This anaconda kickstart file contains the values used to install your server
- We can use this anaconda file to create our kickstart configuration file and configure kickstart server
- You can also use online kickstart generator tool provided by Red Hat.
Create /ks
directory where we will store our kickstart file
[root@rhel-8 pxelinux.cfg]# mkdir /ks
I have created multiple kickstart configuration files for different KVM Virtual Machines on our KVM PXE server
[root@rhel-8 pxelinux.cfg]# cd /ks/
[root@rhel-8 ks]# ls -l total 20 -rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-2.conf -rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-3.conf -rw-r--r--. 1 root root 1352 Apr 30 16:15 kickstart_centos8-4.conf -rw-r--r--. 1 root root 1352 May 4 20:03 kickstart_centos8-5.conf
Step 6.1: Sample kickstart configuration file
Below is my sample kickstart configuration file on the KVM PXE server. I prefer to perform text based network installation and later you can choose to install Graphical packages for GNOME Desktop to your Linux server.
I won't be able to explain the entire content of kickstart file in this article as it would be too long, I have written another article with detailed description of all the values from kickstart file.
#version=RHEL8 # Use text install text # Create new user user --name=deepak --shell=/bin/bash --homedir=/home/deepak --iscrypted --password=$6$uSejt/TeWMJVQ/F8$/oFsIanDHS/5b9ssy7gZbQwNkORgRjsQIw4JyFjlTWDh9TVsEXWEy2APpCUNTHipOOEe..ubg3qBZOwpaPtsB. # Create new repo repo --name="AppStream" --baseurl=file:///run/install/repo/AppStream # Use NFS server for installation nfs --server=192.168.122.1 --dir=/images/ # Keyboard layouts keyboard --vckeymap=us --xlayouts='us' # System language lang en_US.UTF-8 # Network information network --bootproto=static --ip=192.168.122.10 --netmask=255.255.255.0 --gateway=192.168.122.1 --device=eth0 --noipv6 --no-activate network --hostname=centos8-2.example.com # Root password rootpw --iscrypted $6$w7El/FYx9mbTG6x9$Te.Yg6dq0TsQwGpdSjeDGSw4J9ZBAkLXzT9ODMV7I7lHvX3n5.9PCS4jIkS2GbVLZOpVRLvrua3wwbwA.cfWX. # Run the Setup Agent on first boot firstboot --enable # Do not configure the X Window System skipx # System services services --enabled="chronyd" # System timezone timezone Asia/Kolkata --isUtc # Reboot the node post installation reboot # Partition clearing information clearpart --all # Disk partitioning information autopart --type=LVM --fstype=ext4 # List of packages to be installed %packages @Virtualization Host kexec-tools %end # Add on %addon com_redhat_kdump --disable %end
Step 7: Install and Configure NFS
I have already written a separate article with detailed description and steps to install and configure both NFSv4 and NFSv3 server. Hence I will be very brief here:
Install nfs-utils
rpm required to configure NFS on your KVM PXE server
[root@rhel-8 ~]# dnf -y install nfs-utils
- Below are the directories I plan to share for my Linux kickstart server.
- Here
/ks
contains the kickstart configuration file - and
/images
contains the RHEL 8 ISO content for installation
[root@rhel-8 ~]# cat /etc/exports /ks *(ro,sync,no_root_squash) /images *(ro,sync,no_root_squash)
Re-export the shares
[root@rhel-8 ~]# exportfs -r
Print the available shares
[root@rhel-8 ~]# exportfs -v /ks (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,no_root_squash,no_all_squash) /images (sync,wdelay,hide,no_subtree_check,sec=sys,ro,secure,no_root_squash,no_all_squash)
Enable and start the nfs-server.service
. Here since we are using NFSV4, we have not started or installed rpcbind
.
You can read more at: Step-by-Step Guide to install and configure NFSv4 and NFSv3 server & client in RHEL/CentOS 7/8
[root@rhel-8 ~]# systemctl enable nfs-server --now
Make sure nfs-server service up and running
[root@rhel-8 ~]# systemctl is-active nfs-server
active
Step 8: Create KVM VM and perform qemu netboot
- We will use
virt-install
tool to create KVM Virtual Machine - Using
--network
andmac.address
we will assign a custom MAC address to our primary NIC interface to perform qemu netboot of the VM - The MAC address is mapped to our PXE configuration files
- Use
--pxe
to perform network based installation
[root@rhel-8 ~]# virt-install --name centos8-2 --memory 10240 --vcpus=2 --network bridge:virbr0,mac.address=00:17:a4:77:00:45 --pxe --graphics=vnc -v --disk path=/disks/centos8-2.qcow2,size=20 Starting install... Domain installation still in progress. Waiting for installation to complete. Domain has shutdown. Continuing. Domain creation completed. Restarting guest.
Once the above command executes, you can check your system logs using journalctl, as you see we have DHCP messages to release IP, followed by TFTP messages to transfer boot images
Apr 30 15:34:31 rhel-8.example.com dnsmasq-dhcp[664]: DHCPDISCOVER(virbr0) 00:17:a4:77:00:45 Apr 30 15:34:31 rhel-8.example.com dnsmasq-dhcp[664]: DHCPOFFER(virbr0) 192.168.122.170 00:17:a4:77:00:45 Apr 30 15:34:35 rhel-8.example.com dnsmasq-dhcp[664]: DHCPDISCOVER(virbr0) 00:17:a4:77:00:45 Apr 30 15:34:35 rhel-8.example.com dnsmasq-dhcp[664]: DHCPOFFER(virbr0) 192.168.122.170 00:17:a4:77:00:45 Apr 30 15:34:43 rhel-8.example.com dnsmasq-dhcp[664]: DHCPREQUEST(virbr0) 192.168.122.170 00:17:a4:77:00:45 Apr 30 15:34:43 rhel-8.example.com dnsmasq-dhcp[664]: DHCPACK(virbr0) 192.168.122.170 00:17:a4:77:00:45 Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/pxelinux.0 to 192.168.122.170 Apr 30 15:34:47 rhel-8.example.com dnsmasq-script[664]: Unsupported action: tftp Apr 30 15:34:47 rhel-8.example.com dnsmasq[664]: script process exited with status 1 Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/ldlinux.c32 to 192.168.122.170 Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/pxelinux.cfg/01-00-17-a4-77-00-45 to 192.168.122.170 Apr 30 15:34:47 rhel-8.example.com dnsmasq-tftp[664]: sent /var/lib/tftpboot/pxelinux/boot.msg to 192.168.122.170
So our KVM PXE server is running as expected.
Now you don't have to go through so many steps to create KVM Virtual machine every time and your one click install server is ready.
Lastly I hope the steps from the article to configure KVM PXE server on RHEL/CentOS 8 Linux was helpful. So, let me know your suggestions and feedback using the comment section.
Can you reply if its possible to make sth similar like you did but instead using virbr0 as DHCP, make it on bridged network with dedicated dhcp server? I mean I put DHCP, TFTP and NFS server on one VM RHEL 8(VMware) and want to have it as PXE Server and then install from it (nested vm using kvm). However when booting I got message:
Nothing to boot: No such file or directory (http://ipxe.org/2d03e13b)
No more network devices
Address from DHCP is correctly given. Also added to bridge configuration file under libvirt and which looks like the below:
My DHCP conf:
max-lease-time 1200;
default-lease-time 900;
log-facility local7;
option space pxelinux;
option pxelinux.magic code 208 = string;
option pxelinux.configfile code 209 = text;
option pxelinux.pathprefix code 210 = text;
option pxelinux.reboottime code 211 = unsigned integer 32;
option architecture-type code 93 = unsigned integer 16;
subnet 192.168.4.0 netmask 255.255.255.0 {
option routers 192.168.4.2;
option domain-name-servers 127.0.0.1;
range 192.168.4.210 192.168.4.249;
class “pxeclients” {
match if substring (option vendor-class-identifier, 0, 9) = “PXEClient”;
next-server server1.example.com;
filename “pxelinux/pxelinux.0”;
}
}
My
/var/lib/tftpboot/pxelinux/pxelinux.cfg/01-08-00-27-99-A7-5D
:imeout 600
display boot.msg
default 1
prompt 1
label 1
menu label ^Install Red Hat Enterprise Linux 8
kernel vmlinuz
append initrd=initrd.img showopts ks=nfs:192.168.4.200//ks/ks.cfg ip=dhcp
label 2
menu label Boot from ^local drive
localboot 0x80
Image from virt-manager:
https://imgur.com/0jtVmha
Hello,
# systemctl stop firewalld
Do you have any logs from the server side during the PXE stage?
Looks like the DHCP stage has passed and the client got IP 192.168.4.145
So the problem is with TFTP. Few things you can try
1. Disable firewall
2. I see you are using next-server as server1.example.com. I would recommend putting IP address instead unless you have an internal DNS configured which can resolve the hostname
tftp localhost
3. Install tftp on the server and verify if your tftp-server is working
get vmlinuz