In this tutorial we will learn about the new kernel packages which are introduced in RHEL/CentOS 8 i.e kernel-core and kernel-modules. I will also share some tips on best practices which must be followed for updating kernel, especially in production environment.
What is kernel-core and kernel-modules rpm
- The
kernel
RPM is a meta package that does not contain any files, but rather ensures that the following sub-packages are properly installed: kernel-core
contains a minimal number of kernel modules needed for core functionality those which are deemed "necessary" for a system to boot.kernel-modules
- contains further kernel modules which typically would be found in the kernel package in older versions of RHEL.kernel-modules-extra
- contains kernel modules for rare hardware for more obscure hardware and lesser used network scheduling policies and algorithms.
Should we install kernel or kernel-core
- The
kernel
meta-package offers a backwards compatible method of interacting with the kernel packages viarpm
anddnf
. - So in RHEL/CentOS 8, the kernel package doesn't contain any file but is required for backward compatibility
- So it is recommended to install
kernel
package in your environment
Content of kernel rpm
We will query the content of kernel
rpm:
# rpm -ql kernel-4.18.0-193.14.2.el8_2.x86_64
(contains no files)
As you see it prints the same message, i.e. it doesn't contain any file with RHEL/CentOS 8
Content of kernel-core rpm
To list the content of kernel-core
we will use the similar command:
# rpm -ql kernel-core | more
/boot/.vmlinuz-4.18.0-193.el8.x86_64.hmac
/boot/System.map-4.18.0-193.el8.x86_64
/boot/config-4.18.0-193.el8.x86_64
/boot/initramfs-4.18.0-193.el8.x86_64.img
/boot/symvers-4.18.0-193.el8.x86_64.gz
<output trimmed>
As you see this rpm contains the core packages and modules which was earlier part of kernel rpm.
Content of kernel-modules rpm
To list all the modules part of kernel-modules
rpm you can use:
# rpm -ql kernel-modules | grep .xz$
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bcma/bcma.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/ath3k.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bcm203x.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bfusb.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bpa10x.ko.xz
<output trimmed>
You can further use grep in this command to grep for the exact module you are looking for in the kernel-modules rpm
Which kernel packages will I need
- For regular bare metal install, all three subpackage rpms will be installed,
kernel-core
,kernel-modules
,kernel-modules-extra
. - On cloud vendors VMs: only the
kernel-core
will need to be installed.
Best practices to update kernel, kernel-core and kernel-modules
- With RHEL/CentOS 7 we were only concerned about handling of kernel rpm but now we have to look out for
kernel
,kernel-core
andkernel-modules
- We should never use
rpm
with-U
to update the mentioned kernel rpms. - As
rpm -U
is not smart enough to differentiate between a normal package and akernel
rpm so it will treat kernel as any other rpm and remove the older package after the newer one is installed - Now technically this is not a problem but you loose your backup. Assume you did a reboot and your Linux server broke due to kernel upgrade, so now you don't have any backup kernel to boot up your system
- So always always make (yes written 2 times) sure that you always install
kernel
,kernel-core
andkernel-modules
package (and not update) withrpm
command. - Although you can use
dnf
oryum
to update the kernel rpm usingyum update <rpm>
ordnf update <rpm>
- YUM and DNF are smart enough to handle kernel updates and they will always perform an install of
kernel
,kernel-core
andkernel-modules
irrespective of the command you give i.e. install or update - So in production environments always choose to use YUM or DNF when updating kernels.
- If you do not have a repo to be used to YUM or DNF the you can either register your RHEL server or configure an offline repository.
Update kernel, kernel-core and kernel-modules
I am using RHEL 8.1 for this tutorial which is registered with Red Hat Network but I have verified the same steps on CentOS 8 so you can use these steps on either of the nodes.
# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)
List available kernel for upgrade
If you already have downloaded kernel rpms then you can place them in a directory and create offline repository. Since my Linux server is registered with RHN, I will automatically get these updates
To list all the available kernel rpms in the repositories
# dnf --showduplicates list kernel | expand Updating Subscription Management repositories. Last metadata expiration check: 0:00:37 ago on Sat 15 Aug 2020 02:52:02 PM IST. Installed Packages kernel.x86_64 4.18.0-147.el8 @anaconda Available Packages kernel.x86_64 4.18.0-80.el8 rhel-8-for-x86_64-baseos-rpms kernel.x86_64 4.18.0-80.1.2.el8_0 rhel-8-for-x86_64-baseos-rpms kernel.x86_64 4.18.0-80.4.2.el8_0 rhel-8-for-x86_64-baseos-rpms <output_trimmed> kernel.x86_64 4.18.0-193.6.3.el8_2 rhel-8-for-x86_64-baseos-rpms kernel.x86_64 4.18.0-193.13.2.el8_2 rhel-8-for-x86_64-baseos-rpms kernel.x86_64 4.18.0-193.14.3.el8_2 rhel-8-for-x86_64-baseos-rpms
Similarly you can get the list of other kernel rpms i.e. kernel-core
and kernel-modules
kernel
, kernel-core
, kernel-modules
rpm will be same. So any version you choose for kernel, yum
or dnf
will automatically choose the dependency kernel rpms and install them.Currently on my server below list of kernel rpms are installed
# rpm -qa | grep kernel kernel-core-4.18.0-147.el8.x86_64 kernel-modules-4.18.0-147.el8.x86_64 kernel-4.18.0-147.el8.x86_64 kernel-tools-libs-4.18.0-147.el8.x86_64 kernel-tools-4.18.0-147.el8.x86_64
Update kernel, kernel-core, kernel-modules and other dependent packages
Now we will update kernel on our server to 4.18.0-193.el8
version. Since we plan to update the kernel to a specific version, I will also provide the version number. But if you plan to update the kernel to latest available version then you don't need to provide the version details.
# dnf install kernel-4.18.0-193.el8 Updating Subscription Management repositories. Last metadata expiration check: 0:03:28 ago on Sat 15 Aug 2020 03:05:23 PM IST. Dependencies resolved. ====================================================================================================================== Package Architecture Version Repository Size ====================================================================================================================== Installing: kernel x86_64 4.18.0-193.el8 rhel-8-for-x86_64-baseos-rpms 2.8 M Upgrading: kexec-tools x86_64 2.0.20-14.el8 rhel-8-for-x86_64-baseos-rpms 485 k linux-firmware noarch 20191202-97.gite8a0f4c9.el8 rhel-8-for-x86_64-baseos-rpms 72 M Installing dependencies: kernel-modules x86_64 4.18.0-193.el8 rhel-8-for-x86_64-baseos-rpms 23 M kernel-core x86_64 4.18.0-193.el8 rhel-8-for-x86_64-baseos-rpms 28 M <output trimmed> Upgraded: kexec-tools-2.0.20-14.el8.x86_64 linux-firmware-20191202-97.gite8a0f4c9.el8.noarch Installed: kernel-4.18.0-193.el8.x86_64 kernel-modules-4.18.0-193.el8.x86_64 kernel-core-4.18.0-193.el8.x86_64 Complete!
So here I had just given dnf install kernel
, dnf has identified the dependent rpms i.e. kernel-core
and kernel-modules
and has also installed them. If you notice, the kernel rpms are "installed" while some dependent packages are "upgraded"
This is the difference I had highlighted earlier with rpm -U
and DNF/YUM. Now we can list the available kernels after this update
# rpm -qa | grep kernel | sort kernel-4.18.0-147.el8.x86_64 kernel-4.18.0-193.el8.x86_64 kernel-core-4.18.0-147.el8.x86_64 kernel-core-4.18.0-193.el8.x86_64 kernel-modules-4.18.0-147.el8.x86_64 kernel-modules-4.18.0-193.el8.x86_64 kernel-tools-4.18.0-147.el8.x86_64 kernel-tools-libs-4.18.0-147.el8.x86_64
Now I have updated kernel, kernel-core
and kernel-modules
but kernel-tools
and kernel-tools-libs
are still with old version. If you also have any additional kernel rpms then it is recommended to have all the kernel rpms in the same version level
Update kernel-tools and kernel-tools-libs (Optional)
So we will manually update the remaining kernel packages. I have only mentioned kernel-tools
here because I know kernel-tools-libs
is a dependency package and will automatically be chosen by DNF
# dnf update kernel-tools-4.18.0-193.el8
Updating Subscription Management repositories.
Last metadata expiration check: 0:11:19 ago on Sat 15 Aug 2020 03:05:23 PM IST.
Dependencies resolved.
======================================================================================================================
Package Architecture Version Repository Size
======================================================================================================================
Upgrading:
kernel-tools-libs x86_64 4.18.0-193.el8 rhel-8-for-x86_64-baseos-rpms 2.8 M
kernel-tools x86_64 4.18.0-193.el8 rhel-8-for-x86_64-baseos-rpms 2.9 M
<output trimmed>
Upgraded:
kernel-tools-libs-4.18.0-193.el8.x86_64 kernel-tools-4.18.0-193.el8.x86_64
Complete!
Notice here, I have used dnf update <rpm>
instead of install
action. This is because, we need install
action only for core kernel packages. Even if I used install
here, the packages would have upgraded
so doesn't matter
Verify the list of kernel packages installed
# rpm -qa | grep kernel | sort kernel-4.18.0-147.el8.x86_64 kernel-4.18.0-193.el8.x86_64 kernel-core-4.18.0-147.el8.x86_64 kernel-core-4.18.0-193.el8.x86_64 kernel-modules-4.18.0-147.el8.x86_64 kernel-modules-4.18.0-193.el8.x86_64 kernel-tools-4.18.0-193.el8.x86_64 kernel-tools-libs-4.18.0-193.el8.x86_64
So now all our kernel packages are at 4.18.0-193.el8
, some of them are with 4.18.0-147.el8
but we will remove them once we know that our server is working properly.
Check default kernel version
Now since we have updated the kernel rpm, by default the new kernel will be configured to get activated post reboot. To verify this, first check the currently active kernel
# uname -r
4.18.0-147.el8.x86_64
So currently 4.18.0-147.el8
is active. Next check the kernel version which will get activated post reboot
# grubby --grub2 --default-title Red Hat Enterprise Linux (4.18.0-193.el8.x86_64) 8.2 (Ootpa) # grubby --default-kernel /boot/vmlinuz-4.18.0-193.el8.x86_64
Using grubby in RHEL/CentOS 8 we can get to know the kernel which will get activated post reboot. I will write separate article on the usage of grubby and more about switching between different kernel versions in RHEL/CentOS 8
Now since we know the new kernel configuration is properly set to be activated, let's reboot our server. Post reboot, re-verify your default and active kernel version.
Once the node is properly UP, you can verify your application functionality (if any) and if all looks good, we can safely remove the old kernel.
Remove old kernel and kernel-core rpms
I prefer DNF and YUM for installation of kernel rpms but for removal I prefer rpm
command to control the list of rpms I remove as with DNF and YUM, it may delete more number of dependent packages.
So to only remove kernel
, kernel-core
and kernel-modules
of older kernel version we will use
# rpm -e kernel-4.18.0-147.el8.x86_64 kernel-core-4.18.0-147.el8.x86_64 kernel-modules-4.18.0-147.el8.x86_64
List the available kernel versions
# rpm -qa | grep kernel | sort
kernel-4.18.0-193.el8.x86_64
kernel-core-4.18.0-193.el8.x86_64
kernel-modules-4.18.0-193.el8.x86_64
kernel-tools-4.18.0-193.el8.x86_64
kernel-tools-libs-4.18.0-193.el8.x86_64
Now we don't have any more old kernel rpms on our server.
What's Next
In this tutorial I gave you a brief introduction of grubby tool, next let's explore grubby tool and try to switch between different kernel versions in RHEL/CentOS 8
- How to switch between multiple kernel version in RHEL/CentOS 8
- 2 ways to boot with old kernel version in RHEL 8 using grubby
Conclusion
In this tutorial we learned about the do's and dont's of upgrading kernel in real time production environment. kernel is the core of operating system and if it is breaks means your system will be in an unrecoverable state. In such case your only option would be to boot into single user mode and try to fix the issue. So it is always recommended to keep a copy of older kernel on your server until you are sure that the new kernel is compatible with your application.
Lastly I hope the steps from the article to learn about kernel-core and updating kernel on Linux was helpful. So, let me know your suggestions and feedback using the comment section.
References
I have used below external references for this tutorial guide
What are the kernel-core, kernel-modules, and kernel-modules-extras packages?