Best practices to update kernel-core in RHEL/CentOS 8

In this tutorial we will learn about the new kernel packages which are introduced in RHEL/CentOS 8 i.e kernel-core and kernel-modules. I will also share some tips on best practices which must be followed for updating kernel, especially in production environment.

 

What is kernel-core and kernel-modules rpm

  • The kernel RPM is a meta package that does not contain any files, but rather ensures that the following sub-packages are properly installed:
  • kernel-core contains a minimal number of kernel modules needed for core functionality those which are deemed "necessary" for a system to boot.
  • kernel-modules - contains further kernel modules which typically would be found in the kernel package in older versions of RHEL.
  • kernel-modules-extra - contains kernel modules for rare hardware for more obscure hardware and lesser used network scheduling policies and algorithms.

 

Should we install kernel or kernel-core

  • The kernel meta-package offers a backwards compatible method of interacting with the kernel packages via rpm and dnf.
  • So in RHEL/CentOS 8, the kernel package doesn't contain any file but is required for backward compatibility
  • So it is recommended to install kernel package in your environment

Content of kernel rpm

We will query the content of kernel rpm:

# rpm -ql kernel-4.18.0-193.14.2.el8_2.x86_64
(contains no files)

As you see it prints the same message, i.e. it doesn't contain any file with RHEL/CentOS 8

 

Content of kernel-core rpm

To list the content of kernel-core we will use the similar command:

# rpm -ql kernel-core | more
/boot/.vmlinuz-4.18.0-193.el8.x86_64.hmac
/boot/System.map-4.18.0-193.el8.x86_64
/boot/config-4.18.0-193.el8.x86_64
/boot/initramfs-4.18.0-193.el8.x86_64.img
/boot/symvers-4.18.0-193.el8.x86_64.gz
<output trimmed>

As you see this rpm contains the core packages and modules which was earlier part of kernel rpm.

 

Content of kernel-modules rpm

To list all the modules part of kernel-modules rpm you can use:

# rpm -ql kernel-modules | grep .xz$ 
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bcma/bcma.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/ath3k.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bcm203x.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bfusb.ko.xz
/lib/modules/4.18.0-193.el8.x86_64/kernel/drivers/bluetooth/bpa10x.ko.xz
<output trimmed>

You can further use grep in this command to grep for the exact module you are looking for in the kernel-modules rpm

 

Which kernel packages will I need

  • For regular bare metal install, all three subpackage rpms will be installed, kernel-core, kernel-modules, kernel-modules-extra.
  • On cloud vendors VMs: only the kernel-core will need to be installed.

 

Best practices to update kernel, kernel-core and kernel-modules

  • With RHEL/CentOS 7 we were only concerned about handling of kernel rpm but now we have to look out for kernel, kernel-core and kernel-modules
  • We should never use rpm with -U to update the mentioned kernel rpms.
  • As rpm -U is not smart enough to differentiate between a normal package and a kernel rpm so it will treat kernel as any other rpm and remove the older package after the newer one is installed
  • Now technically this is not a problem but you loose your backup. Assume you did a reboot and your Linux server broke due to kernel upgrade, so now you don't have any backup kernel to boot up your system
  • So always always make (yes written 2 times) sure that you always install kernel, kernel-core and kernel-modules package (and not update) with rpm command.
  • Although you can use dnf or yum to update the kernel rpm using yum update <rpm> or dnf update <rpm>
  • YUM and DNF are smart enough to handle kernel updates and they will always perform an install of kernel, kernel-core and kernel-modules irrespective of the command you give i.e. install or update
  • So in production environments always choose to use YUM or DNF when updating kernels.
  • If you do not have a repo to be used to YUM or DNF the you can either register your RHEL server or configure an offline repository.

 

Update kernel, kernel-core and kernel-modules

I am using RHEL 8.1 for this tutorial which is registered with Red Hat Network but I have verified the same steps on CentOS 8 so you can use these steps on either of the nodes.

# cat /etc/redhat-release
Red Hat Enterprise Linux release 8.1 (Ootpa)

 

List available kernel for upgrade

If you already have downloaded kernel rpms then you can place them in a directory and create offline repository. Since my Linux server is registered with RHN, I will automatically get these updates

To list all the available kernel rpms in the repositories

# dnf --showduplicates list kernel | expand
Updating Subscription Management repositories.
Last metadata expiration check: 0:00:37 ago on Sat 15 Aug 2020 02:52:02 PM IST.
Installed Packages
kernel.x86_64        4.18.0-147.el8                @anaconda
Available Packages
kernel.x86_64        4.18.0-80.el8                 rhel-8-for-x86_64-baseos-rpms
kernel.x86_64        4.18.0-80.1.2.el8_0           rhel-8-for-x86_64-baseos-rpms
kernel.x86_64        4.18.0-80.4.2.el8_0           rhel-8-for-x86_64-baseos-rpms

<output_trimmed>

kernel.x86_64        4.18.0-193.6.3.el8_2          rhel-8-for-x86_64-baseos-rpms
kernel.x86_64        4.18.0-193.13.2.el8_2         rhel-8-for-x86_64-baseos-rpms
kernel.x86_64        4.18.0-193.14.3.el8_2         rhel-8-for-x86_64-baseos-rpms

Similarly you can get the list of other kernel rpms i.e. kernel-core and kernel-modules

Currently on my server below list of kernel rpms are installed

# rpm -qa | grep kernel
kernel-core-4.18.0-147.el8.x86_64
kernel-modules-4.18.0-147.el8.x86_64
kernel-4.18.0-147.el8.x86_64
kernel-tools-libs-4.18.0-147.el8.x86_64
kernel-tools-4.18.0-147.el8.x86_64

 

Update kernel, kernel-core, kernel-modules and other dependent packages

Now we will update kernel on our server to 4.18.0-193.el8 version. Since we plan to update the kernel to a specific version, I will also provide the version number. But if you plan to update the kernel to latest available version then you don't need to provide the version details.

# dnf install kernel-4.18.0-193.el8
Updating Subscription Management repositories.
Last metadata expiration check: 0:03:28 ago on Sat 15 Aug 2020 03:05:23 PM IST.
Dependencies resolved.
======================================================================================================================
 Package               Architecture  Version                               Repository                            Size
======================================================================================================================
Installing:
 kernel                x86_64        4.18.0-193.el8                        rhel-8-for-x86_64-baseos-rpms        2.8 M
Upgrading:
 kexec-tools           x86_64        2.0.20-14.el8                         rhel-8-for-x86_64-baseos-rpms        485 k
 linux-firmware        noarch        20191202-97.gite8a0f4c9.el8           rhel-8-for-x86_64-baseos-rpms         72 M
Installing dependencies:
 kernel-modules        x86_64        4.18.0-193.el8                        rhel-8-for-x86_64-baseos-rpms         23 M
 kernel-core           x86_64        4.18.0-193.el8                        rhel-8-for-x86_64-baseos-rpms         28 M

<output trimmed>

Upgraded:
  kexec-tools-2.0.20-14.el8.x86_64                  linux-firmware-20191202-97.gite8a0f4c9.el8.noarch

Installed:
  kernel-4.18.0-193.el8.x86_64      kernel-modules-4.18.0-193.el8.x86_64      kernel-core-4.18.0-193.el8.x86_64

Complete!

So here I had just given dnf install kernel, dnf has identified the dependent rpms i.e. kernel-core and kernel-modules and has also installed them. If you notice, the kernel rpms are "installed" while some dependent packages are "upgraded"

This is the difference I had highlighted earlier with rpm -U and DNF/YUM. Now we can list the available kernels after this update

# rpm -qa | grep kernel | sort
kernel-4.18.0-147.el8.x86_64
kernel-4.18.0-193.el8.x86_64
kernel-core-4.18.0-147.el8.x86_64
kernel-core-4.18.0-193.el8.x86_64
kernel-modules-4.18.0-147.el8.x86_64
kernel-modules-4.18.0-193.el8.x86_64
kernel-tools-4.18.0-147.el8.x86_64
kernel-tools-libs-4.18.0-147.el8.x86_64

Now I have updated kernel, kernel-core and kernel-modules but kernel-tools and kernel-tools-libs are still with old version. If you also have any additional kernel rpms then it is recommended to have all the kernel rpms in the same version level

 

Update kernel-tools and kernel-tools-libs (Optional)

So we will manually update the remaining kernel packages. I have only mentioned kernel-tools here because I know kernel-tools-libs is a dependency package and will automatically be chosen by DNF

# dnf update kernel-tools-4.18.0-193.el8
Updating Subscription Management repositories.
Last metadata expiration check: 0:11:19 ago on Sat 15 Aug 2020 03:05:23 PM IST.
Dependencies resolved.
======================================================================================================================
 Package                     Architecture     Version                   Repository                               Size
======================================================================================================================
Upgrading:
 kernel-tools-libs           x86_64           4.18.0-193.el8            rhel-8-for-x86_64-baseos-rpms           2.8 M
 kernel-tools                x86_64           4.18.0-193.el8            rhel-8-for-x86_64-baseos-rpms           2.9 M

<output trimmed>

Upgraded:
  kernel-tools-libs-4.18.0-193.el8.x86_64                      kernel-tools-4.18.0-193.el8.x86_64

Complete!

Notice here, I have used dnf update <rpm> instead of install action. This is because, we need install action only for core kernel packages. Even if I used install here, the packages would have upgraded so doesn't matter

Verify the list of kernel packages installed

# rpm -qa | grep kernel | sort
kernel-4.18.0-147.el8.x86_64
kernel-4.18.0-193.el8.x86_64
kernel-core-4.18.0-147.el8.x86_64
kernel-core-4.18.0-193.el8.x86_64
kernel-modules-4.18.0-147.el8.x86_64
kernel-modules-4.18.0-193.el8.x86_64
kernel-tools-4.18.0-193.el8.x86_64
kernel-tools-libs-4.18.0-193.el8.x86_64

So now all our kernel packages are at 4.18.0-193.el8, some of them are with 4.18.0-147.el8 but we will remove them once we know that our server is working properly.

 

Check default kernel version

Now since we have updated the kernel rpm, by default the new kernel will be configured to get activated post reboot. To verify this, first check the currently active kernel

# uname -r
4.18.0-147.el8.x86_64

So currently 4.18.0-147.el8 is active. Next check the kernel version which will get activated post reboot

# grubby --grub2 --default-title
Red Hat Enterprise Linux (4.18.0-193.el8.x86_64) 8.2 (Ootpa)

# grubby --default-kernel
/boot/vmlinuz-4.18.0-193.el8.x86_64

Using grubby in RHEL/CentOS 8 we can get to know the kernel which will get activated post reboot. I will write separate article on the usage of grubby and more about switching between different kernel versions in RHEL/CentOS 8

Now since we know the new kernel configuration is properly set to be activated, let's reboot our server. Post reboot, re-verify your default and active kernel version.

Once the node is properly UP, you can verify your application functionality (if any) and if all looks good, we can safely remove the old kernel.

 

Remove old kernel and kernel-core rpms

I prefer DNF and YUM for installation of kernel rpms but for removal I prefer rpm command to control the list of rpms I remove as with DNF and YUM, it may delete more number of dependent packages.

So to only remove kernel, kernel-core and kernel-modules of older kernel version we will use

# rpm -e kernel-4.18.0-147.el8.x86_64 kernel-core-4.18.0-147.el8.x86_64 kernel-modules-4.18.0-147.el8.x86_64

List the available kernel versions

# rpm -qa | grep kernel | sort
kernel-4.18.0-193.el8.x86_64
kernel-core-4.18.0-193.el8.x86_64
kernel-modules-4.18.0-193.el8.x86_64
kernel-tools-4.18.0-193.el8.x86_64
kernel-tools-libs-4.18.0-193.el8.x86_64

Now we don't have any more old kernel rpms on our server.

how to update kernel-core rpm in rhel/centos 8

 

What's Next

In this tutorial I gave you a brief introduction of grubby tool, next let's explore grubby tool and try to switch between different kernel versions in RHEL/CentOS 8

 

Conclusion

In this tutorial we learned about the do's and dont's of upgrading kernel in real time production environment. kernel is the core of operating system and if it is breaks means your system will be in an unrecoverable state. In such case your only option would be to boot into single user mode and try to fix the issue. So it is always recommended to keep a copy of older kernel on your server until you are sure that the new kernel is compatible with your application.

Lastly I hope the steps from the article to learn about kernel-core and updating kernel on Linux was helpful. So, let me know your suggestions and feedback using the comment section.

 

References

I have used below external references for this tutorial guide
What are the kernel-core, kernel-modules, and kernel-modules-extras packages?

Leave a Comment

Please use shortcodes <pre class=comments>your code</pre> for syntax highlighting when adding code.