Grafana→ Loki → Promtail Complete End To End Project
Introduction:
In the previous project we had focused on metrics so we integrated prometheus with grafana but for logging we should use loki and integrate with grafana, loki is like a database of logs, to capture logs we use promtail tool. In this project we will see stack of grafana, loki and promtail because we are focusing on logs instead of metrics calculation.
First we will install grafana in our local system:
Install Grafana on Debian or Ubuntu
sudo apt-get install -y apt-transport-https
sudo apt-get install -y software-properties-common wget
sudo wget -q -O /usr/share/keyrings/grafana.key https://apt.grafana.com/gpg.key
Stable release
echo “deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com stable main” | sudo tee -a /etc/apt/sources.list.d/grafana.list
Beta release
echo “deb [signed-by=/usr/share/keyrings/grafana.key] https://apt.grafana.com beta main” | sudo tee -a /etc/apt/sources.list.d/grafana.list
# Update the list of available packages
sudo apt-get update
# Install the latest OSS release:
sudo apt-get install grafana
#To start Grafana Server
sudo /bin/systemctl status grafana-server
Login to grafana on port 3000 with default user and password “admin”
loki and Promtail using Docker
Make new directory for loki and download config file there:
⇒ mkdir loki
⇒ cd loki
Download Loki Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/cmd/loki/loki-local-config.yaml -O loki-config.yaml
Run Loki Docker container
docker run -d — name loki -v $(pwd):/mnt/config -p 3100:3100 grafana/loki:2.8.0 — config.file=/mnt/config/loki-config.yaml
Check the loki container status:
Download Promtail Config
wget https://raw.githubusercontent.com/grafana/loki/v2.8.0/clients/cmd/promtail/promtail-docker-config.yaml -O promtail-config.yaml
Run Promtail Docker container
docker run -d — name promtail -v $(pwd):/mnt/config -v /var/log:/var/log — link loki grafana/promtail:2.8.0 — config.file=/mnt/config/promtail-config.yaml
Now check the promtail container status:
So we have installed loki and promtail, now we will see how to integrate them with grafana for better visualization.
First add loki as data source of grafana:
Set name and give url of loki then click on save and test:
Once data source is added , then explore it:
And add “varlog” in job section as in conf file of promtail we have define sent all the logs of varlog directory to loki:
When you click on run query you will start seeing all the logs in grafana:
Now we will explore how we can show grafana internal logs via loki:
Now we will define this resource in promtail config file so that i could sent these logs to loki and loki will display those logs on grafana:
Restart the docker container of promtail:
Now grafanalogs will also be visible in grafana:
We can also displayed these logs on grafana dashboard:
Now we will add visualization graph also:
Right now it will be empty,
First install any thing your system, like we install nginx:
Then run query in this visualization that search nginx word in last 5 minutes logs→ then click on visualization suggestions→ then logs:
Now you can see logs which has nginx word in it:
Now we will see how many time nginx repeat for this click on operation:
Thats all in this grafana-loki-promtail end to end project.
— — — — — — — — — —