Grafana→ Loki → Promtail Complete End To End Project

Introduction:

In the previous project we had focused on metrics so we integrated prometheus with grafana but for logging we should use loki and integrate with grafana, loki is like a database of logs, to capture logs we use promtail tool. In this project we will see stack of grafana, loki and promtail because we are focusing on logs instead of metrics calculation.

First we will install grafana in our local system:

Install Grafana on Debian or Ubuntu

sudo apt-get install -y apt-transport-https

sudo apt-get install -y software-properties-common wget

sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key

Stable release

echo “deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main” | sudo tee -a /etc/apt/sources.list.d/grafana.list

Beta release

echo “deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com beta main” | sudo tee -a /etc/apt/sources.list.d/grafana.list

# Update the list of available packages

sudo apt-get update

# Install the latest OSS release:

sudo apt-get install grafana

#To start Grafana Server

sudo /bin/systemctl status grafana-server

Login to grafana on port 3000 with default user and password “admin”

loki and Promtail using Docker

Make new directory for loki and download config file there:

⇒ mkdir loki

⇒ cd loki

Download Loki Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml

Run Loki Docker container

docker run -d — name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 — config.file=/mnt/config/loki-config.yaml

Check the loki container status:

Download Promtail Config

wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml

Run Promtail Docker container

docker run -d — name promtail -v $(pwd):/mnt/config -v /var/log:/var/log — link loki grafana/promtail:2.8.0 — config.file=/mnt/config/promtail-config.yaml

Now check the promtail container status:

So we have installed loki and promtail, now we will see how to integrate them with grafana for better visualization.

First add loki as data source of grafana:

Set name and give url of loki then click on save and test:

Once data source is added , then explore it:

And add “varlog” in job section as in conf file of promtail we have define sent all the logs of varlog directory to loki:

When you click on run query you will start seeing all the logs in grafana:

Now we will explore how we can show grafana internal logs via loki:

Now we will define this resource in promtail config file so that i could sent these logs to loki and loki will display those logs on grafana:

Restart the docker container of promtail:

Now grafanalogs will also be visible in grafana:

We can also displayed these logs on grafana dashboard:

Now we will add visualization graph also:

Right now it will be empty,

First install any thing your system, like we install nginx:

Then run query in this visualization that search nginx word in last 5 minutes logs→ then click on visualization suggestions→ then logs:

Now you can see logs which has nginx word in it:

Now we will see how many time nginx repeat for this click on operation:

Thats all in this grafana-loki-promtail end to end project.

— — — — — — — — — —