In this article we will explore Mutual Transport Layer Security (MTLS) and we will use a client and server setup to quickly validate mTLS authentication. We will use openssl to create the required certificates and verify the mutual TLS authentication.
1. Overview on SSL and TLS
- I hope you are already familiar with SSL and TLS.
- Transport Layer Security (TLS) is a protocol you can use to protect network communications from eavesdropping and other types of attacks.
- It is an update to the Secure Sockets Layer (SSL) protocol that preceded it, and often people still refer to both collectively as “SSL” or use the terms “SSL” and “TLS” interchangeably.
- TLS protects traffic at the transport layer so you can wrap a number of higher-level plain-text protocols in TLS to secure them.
- Some popular examples of protocols that can be wrapped in TLS include HTTP (HTTPS, that lock icon in the URL bar of your browser), FTP (FTPS, not to be confused with SFTP which uses the SSH protocol), IMAP (IMAPS), POP3 (POP3S), and SMTP (SMTPS), among others. As you can see, it’s common to add an “S” at the end of a protocol that is wrapped in SSL or TLS.
TLS provides you with two primary protections.
- The first protection, and the one most commonly associated with TLS, is that TLS encrypts your traffic to protect it from eavesdropping.
- The second and equally important protection is that TLS authenticates the server to the client (and optionally the client to the server as well) with the use of signed certificates.
2. Overview on mTLS
- By default TLS only validates the authenticity of the server and not of the client (application) which is sending the request.
- We can use mutual TLS where both clients request certificates from the server to ensure the server is who it says it is, and the server requests certificates from the client to prove who it is as well.
- When using mutual authentication, not only does the service side prove its identity by exposing a certificate, but also the clients prove their identity to the servers by exposing a client-side certificate.
- This provides a higher level of security compared to normal TLS/HTTPS usage, where only the identity of the server is proven.
- Setting up and maintaining mutual authentication; that is, the provision of new, and the rotating of outdated, certificates, is known to be complex and is therefore seldom used.
3. Why should we use mutual authentication (MTLS)?
Isn't it sufficient to protect external APIs with HTTPS and OAuth 2.0/OIDC access tokens?
As long as the attacks come through the external API, it might be sufficient.
But what if a client node with the application becomes compromised?
For example, if an attacker gains control over the node, then the attacker can start listening to traffic between other nodes in the setup. If the internal communication is sent as plain text, it will be very easy for the attacker to gain access to sensitive information sent between the nodes in the setup. To minimize the damage caused by such an intrusion, mutual authentication can be used to prevent an attacker from eavesdropping on internal network traffic.
4. Lab Environment
I will be using Virtual Machines running on Oracle VirtualBox to demonstrate this article. These VMs are installed with CentOS 7 and 8 Linux. I have two VMs where in one will act as a server while the other will act as a client. I will use the server node to create all the certificates i.e. CA certificate, server and client certificate.
The hostname of the server node is server.example.com with an IP address 192.168.0.114 while the client hostname is server-2.example.com with an IP address of 192.168.0.152
5. Create CA certificate
First we would need a CA certificate which can sign both the client and server certificates. So let's create our directory structure to store the CA certificate and key.
[root@server ~]# mkdir /root/mtls [root@server ~]# cd /root/mtls/ [root@server mtls]# mkdir certs private
Next create an index.txt and serial file to track the list of certificates signed by the CA certificate.
[root@server mtls]# echo 01 > serial [root@server mtls]# touch index.txt
5.1 Configure openssl.cnf
I will copy openssl.cnf from /etc/pki/tls/openssl.cnf and then modify it. I have already explained individual section of this file in Configure openssl.cnf for Root CA Certificate.
So here I will skip the explanation, just sharing a copy of my openssl.cnf for reference:
# This definition stops the following lines choking if HOME isn't
# defined.
HOME = .
RANDFILE = $ENV::HOME/.rnd
# Extra OBJECT IDENTIFIER info:
#oid_file = $ENV::HOME/.oid
oid_section = new_oids
[ new_oids ]
# Policies used by the TSA examples.
tsa_policy1 = 1.2.3.4.1
tsa_policy2 = 1.2.3.4.5.6
tsa_policy3 = 1.2.3.4.5.7
####################################################################
[ ca ]
default_ca = CA_default # The default ca section
[ CA_default ]
dir = /root/mtls # Where everything is kept
certs = $dir/certs # Where the issued certs are kept
database = $dir/index.txt # database index file.
# several certs with same subject.
new_certs_dir = $dir/certs # default place for new certs.
certificate = $dir/certs/cacert.pem # The CA certificate
serial = $dir/serial # The current serial number
crlnumber = $dir/crlnumber # the current crl number
# must be commented out to leave a V1 CRL
private_key = $dir/private/cakey.pem # The private key
name_opt = ca_default # Subject Name options
cert_opt = ca_default # Certificate field options
default_days = 365 # how long to certify for
default_crl_days= 30 # how long before next CRL
default_md = sha256 # use SHA-256 by default
preserve = no # keep passed DN ordering
policy = policy_match
# For the CA policy
[ policy_match ]
countryName = match
stateOrProvinceName = match
organizationName = match
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
[ policy_anything ]
countryName = optional
stateOrProvinceName = optional
localityName = optional
organizationName = optional
organizationalUnitName = optional
commonName = supplied
emailAddress = optional
####################################################################
[ req ]
default_bits = 2048
default_md = sha256
default_keyfile = privkey.pem
distinguished_name = req_distinguished_name
attributes = req_attributes
x509_extensions = v3_ca # The extentions to add to the self signed cert
[ req_distinguished_name ]
countryName = Country Name (2 letter code)
countryName_default = IN
countryName_min = 2
countryName_max = 2
stateOrProvinceName = State or Province Name (full name)
stateOrProvinceName_default = Some-State
localityName = Locality Name (eg, city)
localityName_default = BANGALORE
0.organizationName = Organization Name (eg, company)
0.organizationName_default = GoLinuxCloud
organizationalUnitName = Organizational Unit Name (eg, section)
commonName = Common Name (eg, your name or your server\'s hostname)
commonName_max = 64
emailAddress = Email Address
emailAddress_max = 64
[ req_attributes ]
challengePassword = A challenge password
challengePassword_min = 4
challengePassword_max = 20
unstructuredName = An optional company name
[ v3_req ]
# Extensions to add to a certificate request
basicConstraints = CA:FALSE
keyUsage = nonRepudiation, digitalSignature, keyEncipherment
[ v3_ca ]
# Extensions for a typical CA
subjectKeyIdentifier=hash
authorityKeyIdentifier=keyid:always,issuer
basicConstraints = critical,CA:true
[ crl_ext ]
# issuerAltName=issuer:copy
authorityKeyIdentifier=keyid:always
5.2 Create private key
We would need a private key for the CA certificate.
[root@server mtls]# openssl genrsa -out private/cakey.pem 4096
Generating RSA private key, 4096 bit long modulus
..........................................................++
...........................................++
e is 65537 (0x10001)
5.3 Create CA certificate
We will use this private key to generate our CA certificate:
[root@server mtls]# openssl req -new -x509 -days 3650 -config /root/mtls/openssl.cnf -key private/cakey.pem -out certs/cacert.pem
Sample output snippet from my terminal:
5.4 Convert certificate to PEM format
This is an optional step but you can convert the certificate into PEM format:
[root@server mtls]# openssl x509 -in certs/cacert.pem -out certs/cacert.pem -outform PEM
6. Create client certificate
Now we will create the client certificate which will be used by the client node i.e. server-2.example.com in our case. I will use the same node i.e. server.example.com to generate the client certificates. But let me create a different directory to store these certificates:
[root@server mtls]# mkdir /root/client_certs [root@server mtls]# cd /root/client_certs/
6.1 Create private key
We will again need a different private key for the client certificate.
[root@server client_certs]# openssl genrsa -out client.key.pem 4096
Generating RSA private key, 4096 bit long modulus
..........................................................................++
..................................................................++
e is 65537 (0x10001)
6.2 Generate Certificate Signing Request (CSR)
Next we need to generate the CSR for the client certificate. For our CA certificate we had given few details such as Country Name. State, Locality etc. If you check the openssl.cnf file, the CA certificate is using the policy_match section for the CSR.
policy = policy_match # For the CA policy [ policy_match ] countryName = match stateOrProvinceName = match organizationName = match organizationalUnitName = optional commonName = supplied emailAddress = optional
It is important that any CSR you generate either for client or server certificates, should have matching Country Name, State and Organization Name with the CA certificate or else while signing the certificate you will get error like:
The stateOrProvinceName field needed to be the same in the
CA certificate (XXXXX) and the request (YYYYY
So with that in mind, let's generate the CSR for the client certificate.
[root@server client_certs]# openssl req -new -key client.key.pem -out client.csr
Sample output from my terminal:
6.3 Add certificate extensions
We will also add some certificate extensions to our client certificate. You may modify or ignore this step based on your requirement. We will create an additional configuration file with the required extensions
[root@server client_certs]# cat client_ext.cnf basicConstraints = CA:FALSE nsCertType = client, email nsComment = "OpenSSL Generated Client Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer keyUsage = critical, nonRepudiation, digitalSignature, keyEncipherment extendedKeyUsage = clientAuth, emailProtection
6.4 Create client certificate
Next we will create our client certificate:
[root@server client_certs]# openssl ca -config /root/mtls/openssl.cnf -extfile client_ext.cnf -days 1650 -notext -batch -in client.csr -out client.cert.pem Using configuration from /root/mtls/openssl.cnf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 1 (0x1) Validity Not Before: Apr 8 11:43:21 2021 GMT Not After : Oct 14 11:43:21 2025 GMT Subject: countryName = IN stateOrProvinceName = Karnataka organizationName = GoLinuxCloud commonName = server-2 emailAddress = admin@golinuxcloud.com X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Cert Type: SSL Client, S/MIME Netscape Comment: OpenSSL Generated Client Certificate X509v3 Subject Key Identifier: EA:CD:99:7F:C4:85:DA:A0:16:EF:1C:61:F1:66:8B:13:CD:0D:38:9A X509v3 Authority Key Identifier: keyid:B2:9D:04:DD:06:02:97:93:FC:08:09:CA:B9:F6:7B:13:1B:BB:74:C5 X509v3 Key Usage: critical Digital Signature, Non Repudiation, Key Encipherment X509v3 Extended Key Usage: TLS Web Client Authentication, E-mail Protection Certificate is to be certified until Oct 14 11:43:21 2025 GMT (1650 days) Write out database with 1 new entries Data Base Updated
The path of the CA certificate required to sign the certificate will be picked from /root/mtls/openssl.cnf. You can verify the same using:
[root@server client_certs]# cat /root/mtls/index.txt
V 251014114321Z 01 unknown /C=IN/ST=Karnataka/O=GoLinuxCloud/CN=server-2/emailAddress=admin@golinuxcloud.com
So this says that a certificate for server-2 has been signed by the root CA certificate.
7. Create server certificate
We will again create a separate directory to store the server certificates.
[root@server client_certs]# mkdir ../server_certs [root@server client_certs]# cd ../server_certs/
7.1 Create private key
We need another private key for the server certificate:
[root@server server_certs]# openssl genrsa -out server.key.pem 4096
Generating RSA private key, 4096 bit long modulus
..................++
...............................................................++
e is 65537 (0x10001)
7.2 Create Certificate Signing Request (CSR)
We will follow the same guidelines as I explained while creating CSR for client certificate. The Country, State and Organization Name must match with the CA certificate or else the signing would fail later.
[root@server server_certs]# openssl req -new -key server.key.pem -out server.csr
Sample output snippet from my terminal:
I have highlighted the Common Name used as it is important that you use the hostname/FQDN which will be used to identify the server. Here in my case the FQDN of my server is server.example.com but I have just given the hostname here as the CN value.
7.3 Add certificate extensions
Similar to client certificate, we will again add some extensions to our server certificate. Additionally I am adding Subject Alternative Name field also known as SAN. This is used to define multiple Common Name. In my case it is possible that a client can access the server using IP address or FQDN so I would also like to add those in the certificate.
[root@server server_certs]# cat server_ext.cnf basicConstraints = CA:FALSE nsCertType = server nsComment = "OpenSSL Generated Server Certificate" subjectKeyIdentifier = hash authorityKeyIdentifier = keyid,issuer:always keyUsage = critical, digitalSignature, keyEncipherment extendedKeyUsage = serverAuth subjectAltName = @alt_names [alt_names] IP.1 = 192.168.0.114 IP.2 = 10.0.2.15 DNS.1 = server.example.com
Since I have two interfaces on the server.example.com, I have given the IP address of both the interface as client can also reach to my server using other interface.
7.4 Create server certificate
Now we will use this extension file along with the private key and CSR to generate our server certificate. The path of the CA certificate required to sign the certificate will be picked from /root/mtls/openssl.cnf.
[root@server server_certs]# openssl ca -config /root/mtls/openssl.cnf -extfile server_ext.cnf -days 1650 -notext -batch -in server.csr -out server.cert.pem Using configuration from /root/mtls/openssl.cnf Check that the request matches the signature Signature ok Certificate Details: Serial Number: 2 (0x2) Validity Not Before: Apr 8 11:52:58 2021 GMT Not After : Oct 14 11:52:58 2025 GMT Subject: countryName = IN stateOrProvinceName = Karnataka organizationName = GoLinuxCloud commonName = server emailAddress = admin@golinuxcloud.com X509v3 extensions: X509v3 Basic Constraints: CA:FALSE Netscape Cert Type: SSL Server Netscape Comment: OpenSSL Generated Server Certificate X509v3 Subject Key Identifier: F4:90:6E:30:54:93:35:04:2F:48:DF:D6:55:C8:35:54:4D:29:E6:7E X509v3 Authority Key Identifier: keyid:B2:9D:04:DD:06:02:97:93:FC:08:09:CA:B9:F6:7B:13:1B:BB:74:C5 DirName:/C=IN/ST=Karnataka/L=BANGALORE/O=GoLinuxCloud/CN=ca-server/emailAddress=admin@golinuxcloud.com serial:85:16:E3:D6:C5:59:18:DD X509v3 Key Usage: critical Digital Signature, Key Encipherment X509v3 Extended Key Usage: TLS Web Server Authentication X509v3 Subject Alternative Name: IP Address:192.168.0.114, IP Address:10.0.2.15, DNS:server.example.com Certificate is to be certified until Oct 14 11:52:58 2025 GMT (1650 days) Write out database with 1 new entries Data Base Updated
Verify the index content of the root CA:
[root@server ~]# cat /root/mtls/index.txt
V 251014114321Z 01 unknown /C=IN/ST=Karnataka/O=GoLinuxCloud/CN=server-2/emailAddress=admin@golinuxcloud.com
V 251014115258Z 02 unknown /C=IN/ST=Karnataka/O=GoLinuxCloud/CN=server/emailAddress=admin@golinuxcloud.com
So the root CA has also signed another certificate for CN=server.
8. Validate mutual TLS authentication
Now we will verify the mutual TLS authentication between the server and the client node. To achieve this I will copy the client certificates on to the client node under a new directory /root/certs. I have already created this directory on my client node, here 192.168.0.152 is the IP address of my client i.e. server-2.example.com
[root@server server_certs]# scp ../client_certs/* 192.168.0.152:/root/certs/ root@192.168.0.152's password: client.cert.pem 100% 2220 2.7MB/s 00:00 client.csr 100% 1740 2.4MB/s 00:00 client_ext.cnf 100% 290 580.1KB/s 00:00 client.key.pem 100% 3243 4.9MB/s 00:00
We also need to copy the CA certificate over to the client node:
[root@server server_certs]# scp /root/mtls/certs/cacert.pem 192.168.0.152:/root/certs/ root@192.168.0.152's password: cacert.pem 100% 2106 3.5MB/s 00:00
There are multiple methods to verify the mTLS authentication but I will use nodejs and openssl to validate this. You can also use LDAP, Apache etc to verify this in your environment.
Mehtod-1: Using Nodejs
In this method I will create a simple nodejs file to setup a web server our HTTPS and then access this web server using the client node over TLS.
First make sure nodejs is installed on your server node which is part of EPEL repo:
[root@server ~]# yum install epel-release -y [root@server ~]# yum install nodejs -y
This will install NodeJS on the server node. Next create a simple webserver application file, here I have provided the path of CA certificate, server certificate and server private key.
[root@server ~]# cat web-server.js
const https = require('https');
const fs = require('fs');
const hostname = '192.168.0.114';
const port = 3000;
const options = {
ca: fs.readFileSync('/root/mtls/certs/cacert.pem'),
cert: fs.readFileSync('/root/server_certs/server.cert.pem'),
key: fs.readFileSync('/root/server_certs/server.key.pem'),
rejectUnauthorized: true,
requestCert: true,
};
const server = https.createServer(options, (req, res) => {
res.statusCode = 200;
res.setHeader('Content-Type', 'text/plain');
res.end('Hello World\n');
});
server.listen(port, hostname, () => {
console.log(`Server running at https://${hostname}:${port}/`);
});
Start your web server:
[root@server ~]# node web-server.js
Server running at https://192.168.0.114:3000/
You can also enable port 3000 in the firewall on the server node:
[root@server ~]# firewall-cmd --add-port=3000/tcp --permanent success [root@server ~]# firewall-cmd --reload success
Now login to your client node and using curl try to access the web server:
[root@server-2 ~]# curl --cacert /root/certs/cacert.pem --key /root/certs/client.key.pem --cert /root/certs/client.cert.pem https://192.168.0.114:3000/ Hello World
So as expected, the client was able to connect to the web server using the client certificate. So this proves the mutual TLS authentication where both server and client are using TLS certificate to prove their identity.
Method-2: Using openssl
Let me show you one more method to validate the mutual TLS authentication in Linux using openssl.
On the server node execute the following command:
[root@server ~]# openssl s_server -accept 3000 -CAfile /root/mtls/certs/cacert.pem -cert /root/server_certs/server.cert.pem -key /root/server_certs/server.key.pem -state
Next on the client node execute:
[root@server-2 ~]# openssl s_client -connect 192.168.0.114:3000 -key /root/certs/client.key.pem -cert /root/certs/client.cert.pem -CAfile /root/certs/cacert.pem -state
If everything is correct then you should see a successful TCP handshake between the server and client node.
Sample output from the server node:
[root@server ~]# openssl s_server -accept 3000 -CAfile /root/mtls/certs/cacert.pem -cert /root/server_certs/server.cert.pem -key /root/server_certs/server.key.pem -state
Using default temp DH parameters
ACCEPT
...
-----BEGIN SSL SESSION PARAMETERS-----
MFUCAQECAgMDBALAMAQABDCgaykG+1HPMhTTcfScPIt8DoLAv6WrLYRajaqKCL+J
TXS6zrWEccetgmLAMZf8pZKhBgIEYG9GgqIEAgIBLKQGBAQBAAAA
-----END SSL SESSION PARAMETERS-----
Shared ciphers:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:AES256-GCM-SHA384:AES128-GCM-SHA256:AES256-SHA256:AES128-SHA256:AES256-SHA:AES128-SHA:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA256:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA
Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:0x07+0x08:0x08+0x08:0x09+0x08:0x04+0x08:0x0A+0x08:0x05+0x08:0x0B+0x08:0x06+0x08:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Shared Signature Algorithms: ECDSA+SHA256:ECDSA+SHA384:ECDSA+SHA512:RSA+SHA256:RSA+SHA384:RSA+SHA512:ECDSA+SHA224:RSA+SHA224:ECDSA+SHA1:RSA+SHA1
Supported Elliptic Curve Point Formats: uncompressed:ansiX962_compressed_prime:ansiX962_compressed_char2
Supported Elliptic Curves: 0x001D:P-256:0x001E:P-521:P-384
Shared Elliptic curves: P-256:P-521:P-384
CIPHER is ECDHE-RSA-AES256-GCM-SHA384
Secure Renegotiation IS supported
Sample output from the client node:
[root@server-2 ~]# openssl s_client -connect 192.168.0.114:3000 -key /root/certs/client.key.pem -cert /root/certs/client.cert.pem -CAfile /root/certs/cacert.pem -state
CONNECTED(00000003)
...
SSL handshake has read 4217 bytes and written 415 bytes
Verification: OK
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 4096 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: 7D1BDC137AF9E00707C00FCCFEE195E520FD20EF4C0F0C31BF94E759F3CD0C6E
Session-ID-ctx:
Master-Key: A06B2906FB51CF3214D371F49C3C8B7C0E82C0BFA5AB2D845A8DAA8A08BF894D74BACEB58471C7AD8262C03197FCA592
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
...
Summary
In this tutorial we learned all about Transport Layer Security (MTLS) and how it is different compared to SSL and TLS. We also learned the steps to validate mutual TLS authentication between a server and a client node using openssl and NodeJS. I had written another article to verify similar scenario using Apache server. Now you should be familiar with mTLS and the steps to create and validate certificates.
Further Readings
Implementing SSL/TLS Using Cryptography and PKI
Mutual Authentication
hi,I want to know what if we use the same certificate(thus same private key,same cacert) for both client and servers for tls connection?will it work fine?
also ,in this example you used private key +certificate,is there anyway we can have generate 1 certificate that has both pub key info and private key info with in it?
You can generate certificate with extendedKeyUsage as clientAuth and serverAuth and then we can use the same cert for both client and server. We also use it in production with no issues
Regarding second question, I actually have never tried so can’t give you an honest response. But I will try to test this and write an article soon.