Setup Grafana with Prometheus Rocky Linux 8 [Step-by-Step]

Getting started to Setup Grafana with Prometheus

Welcome to this guide where we shall discuss how to set up Prometheus monitoring tool and Grafana dashboards on Rocky Linux 8. Prometheus is a free open-source monitoring and alerting tool that uses a time-series method to process the data.  Prometheus was initially developed by engineers at SoundCloud but soon gained popularity and became one of the most used monitoring tools in the cloud.

This tool can be used to monitor and alert on systems such as servers, network devices, and application-level metrics such as database performance and web server metrics. You can also configure alerts in case you have any downtimes.

Grafana, on the other hand, is a tool that is used to visualize data from different sources. It provides an interactive interface where one can create dashboards, tweak them, and also import dashboards from the Grafana community.

In this guide, we shall install Prometheus on Rocky Linux 8, we shall also set up a Grafana instance on the same server so that we can visualize metrics populated by the Prometheus service.

First we will install and setup Prometheus on Rocky Linux 8, followed by installation and configuration of Grafana. Follow the steps below to setup Grafana with Prometheus on Rocky Linux 8.

 

Step 1: Create Prometheus System User

We need to create a Prometheus system user and group that will be used to run the application.

sudo groupadd --system prometheus
sudo useradd -s /sbin/nologin --system -g prometheus prometheus

 

Step 2: Create Prometheus Directories

Create the necessary directories for Prometheus. These are the directories that will be used to hold Prometheus data, including the data that will be collected from the agents being monitored.

sudo mkdir /var/lib/prometheus
sudo mkdir /etc/prometheus

 

Step 3: Download Prometheus on Rocky Linux 8

The next step is to download the latest version of Prometheus from Prometheus' GotHub Release page.

You can also download directly on command-line using CURL. This is as shown below:

curl -s https://api.github.com/repos/prometheus/prometheus/releases/latest \
  | grep browser_download_url \
  | grep linux-amd64 \
  | cut -d '"' -f 4 \
  | wget -qi -

Extract the downloaded files and move the binary files to the $PATH of your system.

tar xvf prometheus-*.tar.gz
cd prometheus-*/
sudo cp prometheus promtool /usr/local/bin/

Copy the configuration files to /etc/prometheus/ directory.

sudo cp -r consoles/ console_libraries/ /etc/prometheus/

 

Step 4: Configure Prometheus on Rocky Linux 8

Configure Prometheus on Rocky Linux 8 using the sample file prometheus.yml located in the directory we extracted in the previous step.

Copy the file to /etc/prometheus/

sudo cp prometheus-*/prometheus.yml /etc/prometheus/

This file contains the default configurations for running the Prometheus service. The configuration binds Prometheus service on port 9090 of the Rocky Linux instance.

[user@rocky ~]$ cat /etc/prometheus/prometheus.yml
# Global config
global: 
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.  
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.  
  scrape_timeout: 15s  # scrape_timeout is set to the global default (10s).

# A scrape configuration containing exactly one endpoint to scrape:# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config.
  - job_name: 'prometheus'

    # metrics_path defaults to '/metrics'
    # scheme defaults to 'http'.

    static_configs:
    - targets: ['localhost:9090']

 

Step 5: Set file Permissions for Prometheus

Configure the correct file permissions for Prometheus as shown below:

sudo chown -R prometheus:prometheus /etc/prometheus
sudo chown -R prometheus:prometheus /var/lib/prometheus
sudo chown prometheus:prometheus /usr/local/bin/{prometheus,promtool}

Run the command below to verify that Prometheus configuration is working fine.

sudo prometheus --config.file=/etc/prometheus/prometheus.yml

Confirm that you have an output similar to this below:

[user@rocky ~]$ sudo prometheus --config.file=/etc/prometheus/prometheus.yml
level=info ts=2021-08-31T13:40:41.057Z caller=main.go:390 msg="No time or size retention was set so using the default time retention" duration=15d
level=info ts=2021-08-31T13:40:41.057Z caller=main.go:428 msg="Starting Prometheus" version="(version=2.29.0, branch=HEAD, revision=a3e22d8b3f239e414dd0551708982077fbbd75db)"
level=info ts=2021-08-31T13:40:41.058Z caller=main.go:433 build_context="(go=go1.16.7, user=root@92a842b399bf, date=20210811-06:05:33)"
level=info ts=2021-08-31T13:40:41.058Z caller=main.go:434 host_details="(Linux 5.13.8-1.el8.elrepo.x86_64 #1 SMP Tue Aug 3 12:53:41 EDT 2021 x86_64 rocky (none))"
level=info ts=2021-08-31T13:40:41.059Z caller=main.go:435 fd_limits="(soft=1024, hard=262144)"
level=info ts=2021-08-31T13:40:41.059Z caller=main.go:436 vm_limits="(soft=unlimited, hard=unlimited)"
level=info ts=2021-08-31T13:40:41.068Z caller=web.go:541 component=web msg="Start listening for connections" address=0.0.0.0:9090
level=info ts=2021-08-31T13:40:41.069Z caller=main.go:812 msg="Starting TSDB ..."
level=info ts=2021-08-31T13:40:41.070Z caller=tls_config.go:191 component=web msg="TLS is disabled." http2=false
level=warn ts=2021-08-31T13:40:41.071Z caller=db.go:680 component=tsdb msg="A TSDB lockfile from a previous execution already existed. It was replaced" file=/root/data/lock
level=info ts=2021-08-31T13:40:41.078Z caller=head.go:814 component=tsdb msg="Replaying on-disk memory mappable chunks if any"
level=info ts=2021-08-31T13:40:41.078Z caller=head.go:828 component=tsdb msg="On-disk memory mappable chunks replay completed" duration=4.652µs
level=info ts=2021-08-31T13:40:41.078Z caller=head.go:834 component=tsdb msg="Replaying WAL, this may take a while"
level=info ts=2021-08-31T13:40:41.079Z caller=head.go:891 component=tsdb msg="WAL segment loaded" segment=0 maxSegment=1
level=info ts=2021-08-31T13:40:41.079Z caller=head.go:891 component=tsdb msg="WAL segment loaded" segment=1 maxSegment=1
level=info ts=2021-08-31T13:40:41.079Z caller=head.go:897 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=163.326µs wal_replay_duration=1.029355ms total_replay_duration=1.59288ms
level=info ts=2021-08-31T13:40:41.081Z caller=main.go:839 fs_type=XFS_SUPER_MAGIC
level=info ts=2021-08-31T13:40:41.081Z caller=main.go:842 msg="TSDB started"
level=info ts=2021-08-31T13:40:41.081Z caller=main.go:969 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
level=info ts=2021-08-31T13:40:41.090Z caller=main.go:1006 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=8.742094ms db_storage=727ns remote_storage=1.476µs web_handler=462ns query_engine=851ns scrape=8.184919ms scrape_sd=56.288µs notify=888ns notify_sd=1.319µs rules=1.008µs
level=info ts=2021-08-31T13:40:41.090Z caller=main.go:784 msg="Server is ready to receive web requests."

 

Step 6: Create Systemd Service for Prometheus

Create a systemd service to be able to manage Prometheus using Systemd. Create a file named prometheus.service under /etc/systemd/system/ Then add the content below:

[user@rocky ~]$ cat /etc/systemd/system/prometheus.service
[Unit]
Description=Prometheus
Documentation=https://prometheus.io/docs/introduction/overview/
Wants=network-online.target
After=network-online.target

[Service]
Type=simple
User=prometheus
Group=prometheus
ExecReload=/bin/kill -HUP $MAINPID
ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --web.console.templates=/etc/prometheus/consoles \
  --web.console.libraries=/etc/prometheus/console_libraries \
  --web.listen-address=0.0.0.0:9090 \
  --web.external-url=

SyslogIdentifier=prometheus
Restart=always

[Install]
WantedBy=multi-user.target

Reload the system daemon and start the Prometheus service:

sudo systemctl daemon-reload
sudo systemctl enable --now prometheus

Verify that the service has started and is running:

[user@rocky ~]$ sudo systemctl status prometheus
● prometheus.service - Prometheus
   Loaded: loaded (/etc/systemd/system/prometheus.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-31 16:48:52 EAT; 5min ago
     Docs: https://prometheus.io/docs/introduction/overview/
 Main PID: 1617 (prometheus)
    Tasks: 7 (limit: 6001)
   Memory: 77.7M
   CGroup: /system.slice/prometheus.service
           └─1617 /usr/local/bin/prometheus --config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/pro>

Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.628Z caller=head.go:897 component=tsdb msg="WAL replay completed" checkpoint_replay_duration=4.383526ms wal_replay_duration=378.564401ms>
Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.631Z caller=main.go:839 fs_type=XFS_SUPER_MAGIC
Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.631Z caller=main.go:842 msg="TSDB started"
Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.631Z caller=main.go:969 msg="Loading configuration file" filename=/etc/prometheus/prometheus.yml
Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.640Z caller=main.go:1006 msg="Completed loading of configuration file" filename=/etc/prometheus/prometheus.yml totalDuration=8.785175ms >
Aug 31 16:48:52 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:52.640Z caller=main.go:784 msg="Server is ready to receive web requests."
Aug 31 16:48:58 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:58.921Z caller=compact.go:518 component=tsdb msg="write block" mint=1628776800000 maxt=1628784000000 ulid=01FEE7GQ1JWCMA5C0G8Y5F4CN7 durati>
Aug 31 16:48:58 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:58.925Z caller=head.go:1017 component=tsdb msg="Head GC completed" duration=2.672494ms
Aug 31 16:48:58 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:58.957Z caller=checkpoint.go:97 component=tsdb msg="Creating checkpoint" from_segment=7 to_segment=8 mint=1628784000000
Aug 31 16:48:59 rocky prometheus[1617]: level=info ts=2021-08-31T13:48:59.055Z caller=head.go:1183 component=tsdb msg="WAL checkpoint complete" first=7 last=8 duration=99.225645ms

Check  and verify that Prometheus is running on port 9090

sudo ss -plunt |grep 9090

Expected output:

[user@rocky ~]$ sudo ss -plunt | grep 9090
tcp   LISTEN 0      4096                                    *:9090            *:*    users:(("prometheus",pid=1617,fd=8))

 

Step 7. Configure Firewall to access Prometheus

Configure the firewall to allow Prometheus port.

sudo firewall-cmd --add-port=9090/tcp --permanent sudo 
sudo firewall-cmd --reload

Access the Prometheus web interface at http://server-IP:9090

With our Prometheus instance ready, we can now set up Grafana on Rocky Linux 8.

 

Step 8: Install Grafana On Rocky Linux 8

Before we can install Grafana, we need to configure the Yum repository for the same. To do that, use the command below:

cat <<EOF | sudo tee /etc/yum.repos.d/grafana.repo

[grafana]
name=grafana
baseurl=https://packages.grafana.com/oss/rpm
repo_gpgcheck=1
enabled=1
gpgcheck=1
gpgkey=https://packages.grafana.com/gpg.key
sslverify=1
sslcacert=/etc/pki/tls/certs/ca-bundle.crt
EOF

You can then install Grafana using the YUM command:

sudo dnf install grafana -y

Verify the installed version of Grafana to your Rocky Linux instance:

$ sudo rpm -qi grafana

 

Step 9: Start Grafana service

Start and enable the Grafana service managed by systemd.

$ sudo systemctl enable --now grafana-server.service 

Check that the service has started successfully:

$ sudo systemctl status grafana-server
● grafana-server.service - Grafana instance
   Loaded: loaded (/usr/lib/systemd/system/grafana-server.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2021-08-31 17:22:57 EAT; 1min 45s ago
     Docs: http://docs.grafana.org
 Main PID: 2612 (grafana-server)
    Tasks: 9 (limit: 6001)
   Memory: 97.5M
   CGroup: /system.slice/grafana-server.service
           └─2612 /usr/sbin/grafana-server --config=/etc/grafana/grafana.ini --pidfile=/var/run/grafana/grafana-server.pid --packaging=rpm cfg:default.paths.logs=/var/log/grafana cfg:default.paths.data=/var/lib>

Aug 31 17:22:57 rocky grafana-server[2612]: t=2021-08-31T17:22:57+0300 lvl=info msg="Executing migration" logger=migrator id="add index library_element_connection element_id-kind-connection_id"
Aug 31 17:22:57 rocky grafana-server[2612]: t=2021-08-31T17:22:57+0300 lvl=info msg="Live Push Gateway initialization" logger=live.push_http
Aug 31 17:22:57 rocky systemd[1]: Started Grafana instance.
Aug 31 17:22:57 rocky grafana-server[2612]: t=2021-08-31T17:22:57+0300 lvl=info msg="HTTP Server Listen" logger=http.server address=[::]:3000 protocol=http subUrl= socket=

Grafana service runs on port 3000 by default.

 

Step 10: Configure Firewall for Grafana

Allow Grafana service through the firewall.

sudo firewall-cmd --add-port=3000/tcp --permanent
sudo firewall-cmd --reload

 

Step 11: Access Grafana Dashboard

The Grafana dashboard is accessible through http://server-IP:3000

Access the dashboard using the following default credentials:

username: admin
password: admin

You will be required to set a new password upon the first-time login. This step is optional in the latest versions of Grafana.

After a successful login, you will be redirected to the default dashboard:

 

Step 12: Add Prometheus Data source to Grafana

Click on "Add data source" to add the Prometheus data source and choose Prometheus.

In the Datasource configuration interface, set the HTTP URL to http://localhost:9090

Save the configuration. You can then import dashboards from Grafana.com

With dashboard imported, you should be able to view metrics from your system as shown in the diagram below:

 

Conclusion

In this guide, we discussed how to set up Prometheus and Grafana on Rocky Linux 8. We have gone through all the necessary steps that will get you going with this kind of setup. As discussed, Prometheus gives you an outstanding monitoring solution and has been adopted by many enterprises for systems and application monitoring. Grafana is also a nice tool for the same as it gives you nice dashboards that you can easily rush to, whenever you have issues in your environment. I hope that this guide has been comprehensive enough. Feel free to get in touch in case you run into a challenge during the deployment of these two tools.