This is a multi part Elasticsearch Tutorial where we will cover all the related topics on ELK Stack using Elasticsearch 7.5
- Install and Configure ElasticSearh Cluster 7.5 with 3 Nodes
- Enable HTTPS and Configure SSS/TLS to secure Elasticsearch Cluster
- Install and Configure Kibana 7.5 with SSL/TLS for Elasticsearch Cluster
- Configure Metricbeat 7.5 to monitor Elasticsearch Cluster Setup over HTTPS
- Install and Configure Logstash 7.5 with Elasticsearch
Configure SSL/TLS encryption
When Elasticsearch security is enabled for a cluster that is running with a basic or production license, the use of TLS/SSL for transport communications is obligatory so you must configure SSL/TLS encryption. Additionally, once security has been enabled to secure elasticsearch, all communications to an Elasticsearch cluster must be authenticated, including communications from Kibana and/or application servers.
Elasticsearch has two levels of communications,
- Transport Communications: The transport protocol is used for internal communications between Elasticsearch nodes,
- HTTP Communications: HTTP protocol is used for communications from clients to the Elasticsearch cluster.
Elasticsearch comes with a utility called elasticsearch-certutil
that can be used for generating self signed certificates that can be used to secure elasticsearch for encrypting internal communications within an Elasticsearch cluster.
Create input yml file
We will use a yml file as an input to generate self signed certificates to enable https configuration and secure elasticsearch. You can add more nodes based on your environment
[root@server1 ~]# cat /tmp/instance.yml instances: - name: 'server1' dns: [ 'server1.example.com' ] ip: [ '192.168.0.11' ] - name: "server2" dns: [ 'server2.example.com' ] ip: [ '192.168.0.12' ] - name: 'server3' dns: [ 'server3.example.com' ] ip: [ '192.168.0.13' ] - name: 'centos-8' dns: [ 'centos-8.example.com' ] ip: [ '192.168.0.14' ]
Generate self signed certificate
The elasticsearch-certutil command simplifies the process of generating self signed certificate for the Elastic Stack to enable HTTPS configuration and to secure elasticsearch. It takes care of generating a CA and signing certificates with the CA.
Navigate inside "/usr/share/elasticsearch/
" where we have all the elasticsearch tools
[root@server3 ~]# cd /usr/share/elasticsearch/
Here we will use elasticsearch-certutil
to generate our own self signed certificate to secure elasticsearch. We will store these certificates under /tmp/certs
. If the output directory does not exists, the elasticsearch-certutil
tool will create the same.
[root@server3 elasticsearch]# bin/elasticsearch-certutil cert --keep-ca-key ca --pem --in /tmp/instance.yml --out /tmp/certs/certs.zip <Output trimmed> If you specify any of the following options: * -pem (PEM formatted output) * -keep-ca-key (retain generated CA key) * -multiple (generate multiple certificates) * -in (generate certificates from an input file) then the output will be be a zip file containing individual certificate/key files Directory /tmp/certs does not exist. Do you want to create it? [Y/n]Y Certificates written to /tmp/certs/certs.zip <Output trimmed>
Next navigate inside the output directory /tmp/certs
[root@server3 elasticsearch]# cd /tmp/certs/ [root@server3 certs]# ls certs.zip
Extract the certificates. You will need unzip utility to extract the certificates files
[root@server1 certs]# unzip certs.zip Archive: certs.zip creating: ca/ inflating: ca/ca.crt inflating: ca/ca.key creating: server1/ inflating: server1/server1.crt inflating: server1/server1.key creating: server2/ inflating: server2/server2.crt inflating: server2/server2.key creating: server3/ inflating: server3/server3.crt inflating: server3/server3.key creating: centos-8/ inflating: centos-8/centos-8.crt inflating: centos-8/centos-8.key
Place the certificates
Next to enable HTTPS configuration we will create certs
directory inside /etc/elasticsearch/
on all the cluster nodes to store the self signed certificates
[root@server1 ~]# mkdir -p /etc/elasticsearch/certs [root@server2 ~]# mkdir -p /etc/elasticsearch/certs [root@server3 ~]# mkdir -p /etc/elasticsearch/certs [root@centos-8 ~]# mkdir -p /etc/kibana/certs
Copy the applicable certificate file to /etc/elasticsearch/certs
directory on the localhost
which in our case is server1
[root@server1 ~]# cp /tmp/certs/ca/ca.crt /tmp/certs/server1/* /etc/elasticsearch/certs
Verify the list of files and permissions on these certificate files
[root@server1 certs]# ls -l /etc/elasticsearch/certs total 20 -rw-r--r--. 1 root elasticsearch 1200 Dec 24 22:25 ca.crt -rw-r--r--. 1 root elasticsearch 1196 Dec 24 22:24 server1.crt -rw-r--r--. 1 root elasticsearch 1675 Dec 24 22:24 server1.key
Next copy these certificates to all the elasticsearch cluster nodes in the same location under /etc/elasticsearch/certs
and under /etc/kibana/certs
on centos-8
[root@server1 ~]# scp -r /tmp/certs/ca/ca.crt /tmp/certs/server2/* server2:/etc/elasticsearch/certs/ [root@server1 ~]# scp -r /tmp/certs/ca/ca.crt /tmp/certs/server3/* server3:/etc/elasticsearch/certs/ [root@server1 ~]# scp -r /tmp/certs/ca/ca.crt /tmp/certs/centos-8/centos-8.* centos-8:/etc/kibana/certs/
Enable authentication to secure Elasticsearch
Set xpack.security.enabled
to true
in elasticsearch.yml
of all the elasticsearch cluster nodes to secure elasticsearch and force a custom user authentication for processing any request.
xpack.security.enabled: true
With this to gain access to restricted resources, a user must prove their identity, via passwords, credentials, or some other means (typically referred to as authentication tokens). The Elastic Stack authenticates users by identifying the users behind the requests that hit the cluster and verifying that they are who they claim to be. The authentication process is handled by one or more authentication services called realms.
Enable SSL/TLS to encrypt communication between cluster nodes
The transport protocol is used for communication between nodes to secure Elasticsearch cluster. Because each node in an Elasticsearch cluster is both a client and a server to other nodes in the cluster, all transport certificates must be both client and server certificates.
xpack.security.transport.ssl.enabled: true xpack.security.transport.ssl.key: certs/server1.key xpack.security.transport.ssl.certificate: certs/server1.crt xpack.security.transport.ssl.certificate_authorities: [ "certs/ca.crt" ]
elasticsearch.yml
of all the cluster nodes and change the certificate path and file name accordingly.
Enable HTTPS configuration to encrypt HTTP Client Communications
When security features are enabled, you can optionally use TLS to enable HTTPS configuration and to ensure that communication between HTTP clients and the cluster is encrypted.
xpack.security.http.ssl.enabled: true xpack.security.http.ssl.key: certs/server1.key xpack.security.http.ssl.certificate: certs/server1.crt xpack.security.http.ssl.certificate_authorities: certs/ca.crt
elasticsearch.yml
of all the cluster nodes and change the certificate path and file name accordingly.
Restart Elasticsearch Cluster services
You must perform a full cluster restart to enable HTTPS configuration and secure elasticsearch cluster. Nodes which are configured to use TLS cannot communicate with nodes that are using unencrypted networking (and vice-versa). After enabling TLS you must restart all nodes in order to maintain communication across the cluster.
Check Cluster Status
Now let us try to check the cluster status using API request over HTTPS
[root@server1 ~]# curl --cacert /etc/elasticsearch/certs/ca.crt -XGET https://server1.example.com:9200/_cat/nodes?pretty
{
"error" : {
"root_cause" : [
{
"type" : "security_exception",
"reason" : "missing authentication credentials for REST request [/_cat/nodes?pretty]",
"header" : {
"WWW-Authenticate" : [
"Bearer realm="security"",
"ApiKey",
"Basic realm="security" charset="UTF-8""
]
}
}
],
--cacert
to avoid any certificate related warning as we are using a self signed certificate. You can avoid --cacert
directive if you are using a certificate from a registered CAAs you see the cluster API request fails due to missing authentication. Since we have enabled security features to secure elasticsearch, we will use username password of either any built in user or a file user to authorise the API request. Although there are other methods using which we can authenticate the request but that is not in the scope of this article.
Change built-in user's password
By default we do not know the default password of the built-in users. So we will change the password of all the built-in user using elasticsearch-setup-passwords
. This tool is available under /usr/share/elasticsearch
. Navigate under this path and execute the command bin/elasticsearch-setup-passwords auto
. This will generate random passwords for the various internal stack users.
You can alternatively skip the auto
parameter and manually define your passwords using the interactive
parameter. Keep track of these passwords, we’ll need them again soon.
[root@server1 ~]# /usr/share/elasticsearch/bin/elasticsearch-setup-passwords interactive Initiating the setup of passwords for reserved users elastic,apm_system,kibana,logstash_system,beats_system,remote_monitoring_user. You will be prompted to enter passwords as the process progresses. Please confirm that you would like to continue [y/N]y Enter password for [elastic]: Reenter password for [elastic]: Enter password for [apm_system]: Reenter password for [apm_system]: Enter password for [kibana]: Reenter password for [kibana]: Enter password for [logstash_system]: Reenter password for [logstash_system]: Enter password for [beats_system]: Reenter password for [beats_system]: Enter password for [remote_monitoring_user]: Reenter password for [remote_monitoring_user]: Changed password for user [apm_system] Changed password for user [kibana] Changed password for user [logstash_system] Changed password for user [beats_system] Changed password for user [remote_monitoring_user] Changed password for user [elastic]
Here I have given custom password to all the built in users.
Check elasticsearch cluster health status
Now I can use elastic
user to check the cluster health
[root@server1 ~]# curl --cacert /etc/elasticsearch/certs/ca.crt -u elastic -XGET https://server1.example.com:9200/_cat/nodes?pretty Enter host password for user 'elastic': 192.168.0.12 8 96 0 1.00 1.01 1.05 dim - server2 192.168.0.13 21 93 2 1.02 1.03 1.05 dim * server3 192.168.0.11 29 83 1 1.00 1.01 1.05 dim - server1
As you see we were able to enable https configuration using self signed certificate to secure elasticsearch cluster.
Troubleshoot error messages
Below are some of the error scenarios I faced while trying to configure ELK Stack
Error: failed to initialize SSL TrustManager
Caused by: org.elasticsearch.ElasticsearchException: failed to initialize SSL TrustManager - access to read truststore file [/some/path/certs/elastic-stack-ca.p12] is blocked; SSL resources should be placed in the [/etc/elasticsearch] directory
Explanation:
As per the official elastic guide we can place the certificates under any location to enable https configuration and secure elasticsearch but for some reason with elasticsearch 7.5 I was getting the above error when I place the certificates under my home folder.
Solution:
The recommendation here was to place the certificates under /etc/elasticsearch
. You can create a certs
directory and place the certificates there.
Error: failed to decrypt safe contents entry
Caused by: java.security.UnrecoverableKeyException: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: Given final block not properly padded. Such issues can arise if a bad key is used during decryption.
Explanation:
If you have provided a password for the CA and other certificates then those needs to be added the respective keystore, here since we are configuring SSL/TLS for Elasticsearch cluster hence we must add those passwords to elasticsearch keystore.
How to Fix:
You can follow this guide for more details to add password to elasticsearch keystore for PEM and PKCS#12 format
Lastly I hope the steps from the article to enable SSL and HTTPS configuration using self signed certificate for encrypted communication to secure elasticsearch cluster on Linux was helpful. So, let me know your suggestions and feedback using the comment section.
” : “security_exception”, “reason” : “missing authentication credentials for REST request [/_cat/nodes?pretty]”,