In this article, we will examine the basics of patch management, how you can apply security hotfix and patches to your computer in LIVE production environment, and where you should get patches for RHEL Linux distributions. Patch management methods vary by distribution. If you’re paying for support from Red Hat or SUSE, you’re paying in part for support through their patch management systems.
What Is a Patch?
A patch is an update. It incorporates changes in source code. Patches are normally applied to specific software components, such as the kernel, or a service, such as vsFTP. Patches may fix bugs, address security issues, or incorporate new features.
Identifying security vulnerabilities
From time to time multiple security related vulnerabilities are reported on Linux platform. If you’ve paid for a subscription to a Red Hat or a SUSE distribution, you can get email warnings about security problems with your installed services.
You can use below pages to get the globally identified vulnerabilities and CVE (Common Vulnerabilities and Exposures) list
Now there are 1000s of vulnerabilities identified on a daily basis so it is not possible for anyone to check these vulnerabilities list individually and then map them to your environment. Hence we perform security scan on your Linux box to identify the list of vulnerabilities which are impacting our system and relevant to our product.
There are a number of tools available which can be used to scan your Linux environment, some of them are
We in our environment have used Nessus and Accunetix to scan our Linux system to identify all the vulnerabilities.
Sample Nessus report
Here I cannot put the entire report due to contractual reasons but I have put some snippets from the report which shows the list of vulnerable rpms and CVE
Below table shows the list of impacted CVE with their description for kernel rpm which is installed on my RHEL Linux system:
This description list is followed by the list of CVE. This is just a short output what was there in the report
Below table shows the vulnerable kernel rpm and the one which we should install to mitigate all the vulnerabilities.
There are several sources for patches and upgrades. The best source is generally the upgrade repository pre configured for your distribution. However, there is often a delay when distribution developers process updates from other sources, such as the kernel, or services, such as the Apache Web server.
Depending upon your support contract you can request the developers to prioritise the patch delivery timelines. If you’re in a hurry, you can download packages from the Web site directly associated with your service. While not built for your distribution, it can help you get new features into service as quickly as possible.
Now depending upon your environment you can choose for online patch source or an offline patch source. We will discuss both these topics in depth later in this article.
Perform Patch Management (Apply Security HotFix)
Ideally patch management refers when you have been reported with a mission critical vulnerability (since this article is all about security fixes we will consider vulnerability as our primary root cause) and customer is requesting for an immediate fix.
In such case you may deliver a small hotfix which will apply the patch on all the nodes in customer environment. Now this security hotfix can work online as well as offline.
Online patch management
To use online patch management your RHEL Linux system must be registered with Red Hat Network mapped with proper subscription channel to get the required updates. If you have a substantial number of Linux computers, it may be cost effective to buy, configure, and dedicate one or more computers to the patch management task. For example, assume that you have a network of 100 computers, and patch management requires that each of these computers downloads 20MB per day. Downloading an additional 2GB per day, every day, can be expensive on business-level Internet connections.
In RHEL 7 and 8 this can be achieved using
yum-security plugin, for RHEL 6 you must install
yum-plugin-security rpm manually
On RHEL 6
# yum install yum-plugin-security
List Available Erratas
To list all available erratas without installing them, run:
# yum updateinfo list available RHSA-2014:1031 Important/Sec. 389-ds-base-18.104.22.168-26.el7_0.x86_64 RHSA-2015:0416 Important/Sec. 389-ds-base-22.214.171.124-13.el7.x86_64 RHBA-2015:0626 bugfix 389-ds-base-126.96.36.199-15.el7_1.x86_64 RHSA-2015:0895 Important/Sec. 389-ds-base-188.8.131.52-16.el7_1.x86_64 RHBA-2015:1554 bugfix 389-ds-base-184.108.40.206-20.el7_1.x86_64 RHBA-2015:1960 bugfix 389-ds-base-220.127.116.11-23.el7_1.x86_64 RHBA-2015:2351 bugfix 389-ds-base-18.104.22.168-19.el7.x86_64 <Output trimmed>
List and Install with Security Updates
To list all available security updates without installing them, run:
# yum updateinfo list security all RHSA-2018:3056 Moderate/Sec. samba-client-4.8.3-4.el7.x86_64 RHSA-2019:2099 Moderate/Sec. samba-client-4.9.1-6.el7.x86_64 i RHSA-2016:0006 Moderate/Sec. samba-client-libs-4.2.3-11.el7_2.x86_64 i RHSA-2016:0448 Moderate/Sec. samba-client-libs-4.2.3-12.el7_2.x86_64 i RHSA-2016:0612 Critical/Sec. samba-client-libs-4.2.10-6.el7_2.x86_64 <Output trimmed>
# yum updateinfo list sec i RHSA-2014:0678 Important/Sec. kernel-3.10.0-123.1.2.el7.x86_64 i RHSA-2014:0786 Important/Sec. kernel-3.10.0-123.4.2.el7.x86_64 i RHSA-2014:0923 Important/Sec. kernel-3.10.0-123.4.4.el7.x86_64 i RHSA-2014:1023 Important/Sec. kernel-3.10.0-123.6.3.el7.x86_64 i RHSA-2014:1281 Moderate/Sec. kernel-3.10.0-123.8.1.el7.x86_64 <Output trimmed>
To get a list of the currently installed security updates this command can be used:
# yum updateinfo list security installed RHSA-2014:0678 Important/Sec. kernel-3.10.0-123.1.2.el7.x86_64 RHSA-2014:0786 Important/Sec. kernel-3.10.0-123.4.2.el7.x86_64 RHSA-2014:0923 Important/Sec. kernel-3.10.0-123.4.4.el7.x86_64 RHSA-2014:1023 Important/Sec. kernel-3.10.0-123.6.3.el7.x86_64 <output trimmed>
To know more about the advisory before you apply it:
[root@rhel-fews-cc ~]# yum updateinfo RHSA-2019:2135 Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager =============================================================================== Moderate: qt5 security, bug fix, and enhancement update =============================================================================== Update ID : RHSA-2019:2135 Release : 0 Type : security Status : final Issued : 2019-08-06 08:04:56 UTC Updated : 2019-08-06 08:04:44 UTC Bugs : 1564000 - Rebase qt5-qtbase to 5.9.7 : 1564001 - Rebase qt5-qtcanvas3d to 5.9.7 : 1564002 - Rebase qt5-qtconnectivity to 5.9.7 : 1564003 - Rebase qt5-qtdeclarative to 5.9.7 : 1564004 - Rebase qt5-qtdoc to 5.9.7 : 1564006 - Rebase qt5-qtgraphicaleffects to 5.9.7 : 1564007 - Rebase qt5-qtimageformats to 5.9.7 : 1564008 - Rebase qt5-qtlocation to 5.9.7 <output trimmed> : refer to the CVE page(s) listed in the References : section. : : Additional Changes: : : For detailed information on changes in this : release, see the Red Hat Enterprise Linux 7.7 : Release Notes linked from the References section. Severity : Moderate updateinfo info done <Output trimmed>
If you want to apply only one specific advisory:
# yum update --advisory=RHSA-2014:0159
To list all available security updates with verbose descriptions of the issues they apply to:
# yum info-sec =============================================================================== GeoIP bug fix and enhancement update =============================================================================== Update ID : RHBA-2019:2224 Release : 0 Type : bugfix Status : final Issued : 2019-08-06 08:14:36 UTC Updated : 2019-08-06 08:14:34 UTCDescription : GeoIP is a C library that enables the user to find the country : that any IP address or host name originates from. : It uses a file-based database that can be, : optionally, updated on a weekly basis by : installing the GeoIP-update package. : : For detailed information on changes in this : release, see the Red Hat Enterprise Linux 7.7 : Release Notes linked from the References section. : : Users of GeoIP are advised to upgrade to these : updated packages. Severity : None <Output trimmed>
List and Install Vulnerability with CVE
To view CVEs which affect the system with:
# yum updateinfo list cves CVE-2018-14633 Moderate/Sec. kernel-3.10.0-957.1.3.el7.x86_64 CVE-2018-14646 Moderate/Sec. kernel-3.10.0-957.1.3.el7.x86_64 CVE-2018-18397 Important/Sec. kernel-3.10.0-957.5.1.el7.x86_64 CVE-2018-18559 Important/Sec. kernel-3.10.0-957.5.1.el7.x86_64 CVE-2018-9568 Important/Sec. kernel-3.10.0-957.10.1.el7.x86_64 CVE-2018-17972 Important/Sec. kernel-3.10.0-957.10.1.el7.x86_64 <utput trimmed>
To install packages impacting a certain CVE Number
# yum update --cve CVE-2008-0947
Update all available security updates
Run the following command to download and apply all available security updates from Red Hat Network hosted or Red Hat Network Satellite:
# yum -y update --security
To only install the packages that have a security errata use
# yum update-minimal --security -y
Offline Patch Management
Most of the production environment are not connected to Internet hence online patch management is not possible. So in such cases there are two possible approach taken by customers.
- Create a local repo which is always in synch with Red Hat Network. If you can configure a proxy server, you could download Linux patch data once from the Internet, and then the 100 computers on your network could download the patches locally. You would then save the additional costs for your Internet connection.
- Create a security hotfix with all the packages. This hotfix will contain scripts to create local repo and update the packages locally on individual nodes or on some HTTP server. With this you do not need access to external network in your production environment and is the most secure method to perform patch management and apply security hotfix.
Steps to create offline security hotfix
Based on Nessus scan report you will get the list of CVE or vulnerabilities which are impacting your Linux node. So you can download the rpms which fixes the respective CVE as explained under Online Patch Management
Place all the rpms under one location on any Linux node, for example in our case we will keep all the rpms under
# mkdir /tmp/rhel_security_updates
Next once you have the list of rpms which you need to download, you can then download these rpms from RHN along with their dependencies and keep it under the same path
createrepo as shown below
# cd /tmp/rhel_security_updates # createrepo .
This will create the necessary
repodata files required to create an offline repo
Now our repo directory is ready for offline security hotfix. You can create a script which can now do the below list of tasks
- Create repo file required to create a repo on individual node. A sample content is placed below
[rhel74_updates] name=rhel74_updates baseurl=file:///tmp/rhel_security_updates gpgcheck=1 enabled=1 gpgkey = file:///etc/pki/rpm-gpg/RPM-GPG-KEY-redhat-release
- Update all the rpms using (
yum update -y)
- Verify if the update was successful
Perform Rollback or Fallback after applying security hotfix
Now it is very important that in production environment you have an option available to rollback or fallback. So to fallback or rollback to the older working state of your RHEL Linux host you must know the last state.
Here I mean that assuming the existing (before applying patch or security hotfix) your RHEL system had 500 rpms then after fallback it is expected that you return the system to same set with 500 rpms.
Now before applying security hotfix I had below list of kernel rpms so after fallback also I should have same set of rpms
# rpm -qa | grep kernel kernel-tools-libs-3.10.0-957.21.3.el7.x86_64 kernel-3.10.0-957.21.3.el7.x86_64 kernel-tools-3.10.0-957.21.3.el7.x86_64
This is really tricky to handle because with every security update you install on your RHEL Linux host, there are a number of dependencies and additional rpms which gets installed.
Now for example I wish to update
samba-common rpm due to some vulnerability. Currently my RHEL system has
# rpm -qa | grep samba samba-common-libs-4.8.3-4.el7.x86_64 samba-common-4.8.3-4.el7.noarch samba-client-libs-4.8.3-4.el7.x86_64
While there is a security update available with
# yum updateinfo list sec | grep samba RHSA-2019:2099 Moderate/Sec. samba-client-libs-4.9.1-6.el7.x86_64 RHSA-2019:2099 Moderate/Sec. samba-common-4.9.1-6.el7.noarch RHSA-2019:2099 Moderate/Sec. samba-common-libs-4.9.1-6.el7.x86_64
So I plan to update
# yum update samba-common <Output trimmed> Dependencies Resolved ============================================================================================= Package Arch Version Repository Size ============================================================================================= Updating: samba-common noarch 4.9.1-6.el7 rhel-7-server-rpms 209 k Updating for dependencies: libsmbclient x86_64 4.9.1-6.el7 rhel-7-server-rpms 137 k libtevent x86_64 0.9.37-1.el7 rhel-7-server-rpms 40 k libwbclient x86_64 4.9.1-6.el7 rhel-7-server-rpms 111 k samba-client-libs x86_64 4.9.1-6.el7 rhel-7-server-rpms 4.9 M samba-common-libs x86_64 4.9.1-6.el7 rhel-7-server-rpms 170 k
As you see due to dependency reason I was supposed to also update additional rpms. But there is no such guarantee that while performing downgrade of
samba-common to 4.8.3-4.el7.noarch we will get the same set of dependency list.
In such cases you can manually download the individual rpm → identify the dependencies and then downgrade the rpm using “
rpm” command. But this is very hectic and not recommended.
I recommend using LVM Snapshot feature to perform fallback of such security hotfix. In such case if you wish to fallback then you can just revert back the using the LVM snapshot.
Starting RHEL 7.7 and RHEL 8 you can also boot your RHEL system using the LVM snapshot using BOOM.
It is the most reliable solution for such use cases. Although to perform LVM snapshot you need some mandatory prerequisites which I have explained in detail in a separate article.
Lastly I hope the steps from the article to get an overview on applying security hotfix and performing patch management on RHEL Linux was helpful. So, let me know your suggestions and feedback using the comment section.