Chainlink Prometheus/Grafana TLS monitoring

Indroduction

This documentation is a guide for full Chainlink node monitoring and alerting including the following deployments:

• Prometheus server with TLS & basic-auth
• Prometheus node exporter with TLS & basic-auth
• Grafana with TLS & basic-auth
• Loki & Promtail
• Full monitoring Chainlink dashboard
• Alerts and Telegram notification setup

Comment

• For the example deployment we used the created "Kovan" docker network. Every container needs to be in the same network like the Chainlink node to ensure the communication between them.
• For the creation of the files we used nano, you can also do it with vim
• You need to copy the files from this repository to your system. Just copy the source code after you've created the file by following the guide.

Create directories

Create the directories for all necessary files

mkdir ~/.monitoring
mkdir ~/.monitoring/.tls
mkdir ~/.monitoring/.tls/.prometheus
mkdir ~/.monitoring/.tls/.grafana
mkdir ~/.monitoring/.tls/.node-exporter

TLS certificates

The TLS certificates are created via openssl and stored in the created directories

Prometheus

cd ~/.monitoring/.tls/.prometheus && openssl req -x509 -out   ~/.monitoring/.tls/.prometheus/prometheus.crt  -keyout  ~/.monitoring/.tls/.prometheus/prometheus.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Node exporter

cd ~/.monitoring/.tls/.node-exporter && openssl req -x509 -out   ~/.monitoring/.tls/.node-exporter/node-exporter.crt  -keyout  ~/.monitoring/.tls/.node-exporter/node-exporter.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Grafana

cd ~/.monitoring/.tls/.grafana && openssl req -x509 -out   ~/.monitoring/.tls/.grafana/grafana.crt  -keyout  ~/.monitoring/.tls/.grafana/grafana.key -newkey rsa:2048 -nodes -sha256 -days 365 -subj '/CN=localhost' -extensions EXT -config <( printf "[dn]\nCN=localhost\n[req]\ndistinguished_name = dn\n[EXT]\nsubjectAltName=DNS:localhost\nkeyUsage=digitalSignature\nextendedKeyUsage=serverAuth")

Authentication

A .htpasswd file is used for the protection of the Prometheus credentials using HTTP authentication and is implemented into a .htaccess file.

Install HTPASSWD

yum install httpd-tools

or

sudo apt-get install httpd-tools

Prometheus auth

htpasswd -nBC 10 "" | tr -d ':\n'

you need to save this value for the prometheusweb.yml

Node exporter auth

htpasswd -nBC 10 "" | tr -d ':\n'

you need to save this value for the exporterweb.yml

Node exporter

Create web.yml

cd ~/.monitoring && nano exportweb.yml

copy the code of the exportweb.yml and just change the username of the basic-auth and the HTPASWD token

Run node-exporter

cd ~/.monitoring && docker run -d -p 9100:9100 --name node-exporter --restart unless-stopped --network kovan --user root -v "/:/hostfs" -v /home/<USER>/.monitoring/exporterweb.yml:/hostfs/web.yml -v /home/<USER>/.monitoring/.tls/node-exporter.key:/tls/node-exporter.key -v /home/<USER>/.monitoring/.tls/node-exporter.crt:/tls/node-exporter.crt prom/node-exporter --path.rootfs=/hostfs --web.config=/hostfs/web.yml

You need to change the to your user name in order to gain access. This will point the initialisation to the created and required files and directories.

Prometheus server

Create web.yml

cd ~/.monitoring && nano prometheusweb.yml

Create prometheus.yml

cd ~/.monitoring && nano prometheus.yml

Run prometheus-server

cd ~/.monitoring && sudo docker run --name prometheus --network kovan --restart=unless-stopped --user root -d -p 9090:9090 -v /home/<USER>/.monitoring/prometheus.yml:/etc/prometheus/prometheus.yml -v /home/<USER>/.monitoring/.tls/prometheus.key:/tls/prometheus.key -v /home/<USER>/.monitoring/.tls/prometheus.crt:/tls/prometheus.crt -v /home/<USER>/.monitoring/prometheusweb.yml:/etc/prometheus/web.yml prom/prometheus --config.file=/etc/prometheus/prometheus.yml --web.config.file=/etc/prometheus/web.yml

You need to change the to your user name in order to gain access. This will point the initialisation to the created and required files and directories.

To check if Prometheus scrapes all metrics, you need to check your targets in the Prometheus GUI: https://localhost:9090/targets

Loki

Create loki.yml

cd ~/.monitoring && nano loki.yml

Run Loki

cd ~/.monitoring && sudo docker run -d -p 3100:3100 --name loki --network kovan --restart unless-stopped -v /home/<USER>/.monitoring/loki.yml:/mnt/config/loki.yml grafana/loki:2.2.0 -config.file=/mnt/config/loki.yml

Promtail

Create promtail.yml

cd ~/.monitoring && nano promtail.yml

Run promtail

cd ~/.monitoring && sudo docker run -d --name promtail --network kovan --restart unless-stopped --user root -v /home/<USER>/.monitoring/promtail.yml:/mnt/config/promtail.yml -v /var/log:/var/log -v /var/lib/docker:/var/lib/docker grafana/promtail:2.2.0 -config.file=/mnt/config/promtail.yml

Grafana

Create default.ini

cd ~/.monitoring && nano grafana.ini

Run Grafana

cd ~/.monitoring && docker run -d -p 3000:3000 --name grafana --network kovan --restart unless-stopped --user root -v /home/<USER>/.monitoring/.tls/.grafana/grafana.key:/tls/grafana.key -v /home/<USER>/.monitoring/.tls/.grafana/grafana.crt:/tls/grafana.crt -v /home/<USER>/.monitoring/grafana.ini:/etc/grafana/grafana.ini -e GF_PATHS_CONFIG=/etc/grafana/grafana.ini grafana/grafana:latest

Data source Integration

• Open your Grafana GUI in your explorer https://localhost:3000
• Fill in your username and password
• Add a new data source

Prometheus

• Target: https://<PROMETHEUS_CONTAINER_ID>:9090
• Enable: Basic_Auth, Credentials , CA_Cert, Skip_TLS_VERIFY
• SAVE & TEST

LOKI

• Target: http://<LOKI_CONTAINER_ID>:3100
• SAVE & TEST

Grafana dashboards

Chainlink dashboard:

• Click on Create -> Import
• Import via panel JSON
• Paste the JSON code from this repo's dashboard.file

Host dashboard:

• Click on Create -> Import

• Import via grafana.com

• Type in: 11952

• https://grafana.com/grafana/dashboards/11952

  

Alerting

Create a notification channel:

• "Alerting" -> "Notification channels" -> "Add channel"
• Name: Telegram
• Type: Telegram
• BOT API TOKEN: You need to create A BOT API Token https://medium.com/shibinco/create-a-telegram-bot-using-botfather-and-get-the-api-token-900ba00e0f39
• Chat ID: get the chat ID of your Telegram channel

Alerting

You can now set alerts on your dashboard. You can only set alerts on "graph-visualisations" as displayed metrics.

For a list of important alerts for a running Chainlink node you can have a look at our security research paper: https://linkriver.io/wp-content/uploads/2021/03/Chainlink_Node_Operations_Research_Paper.pdf