self-host/services/grafana/README.md

Local development

docker-compose-up -d

Check which containers are connected to the Docker network:

docker network inspect web

List networks:

docker network ls

Using new Docker Compose syntax:

docker compose up -d

docker compose --env-file .env.prd up -d

Prometheus

The tool that serves as the data source for the Grafana representation layer. Adds timestamps to enable series data. Includes query language (PromQL) that Grafana uses to retrieve and transform data. Also provides alerting.

In a typical implementation, Prometheus collects the raw metrics data from various systems, and Grafana connects to Prometheus as a data source to create dashboards with visualizations of those metrics.

Node exporter

Prometheus has different plug-ins called "exporters". These expose metrics from different processes and services and present them in a format that Prometheus can scrape. Node Exporter is one of these - it exposes hardware and OS metrics (such as CPU use and load, memory, disk capacity etc).

Loki

Loki is for storing and displaying logs. This is part of Grafana. Promtail (also part of Grafana) is for collecting the logs.

Explaining reverse proxy

Yes, that's correct. Let me break down how the overall system works at the top level:

Nginx (Your Reverse Proxy)

Listens on public ports 80 (HTTP) and 443 (HTTPS) Has separate config files in conf.d/ for each service Each config file (like grafana.conf) specifies:

The domain name (grafana.systemsobscure.net) HTTPS certificate configuration Redirection from HTTP to HTTPS Which internal service and port to route requests to (e.g., grafana:3000)

Docker Networks

A shared external network (typically called "web") connects Nginx to all your services Each service also has its own internal network for service-specific communication

Services (Grafana, etc.)

Run on their own internal ports (Grafana on 3000, etc.) Connect to the shared "web" network so Nginx can reach them Are NOT directly exposed to the internet

The flow works like this:

User requests grafana.systemsobscure.net Request hits your server on port 80/443 Nginx receives it, finds the matching server_name in conf.d Nginx forwards the request to the internal Grafana container on port 3000 Response flows back through the same path

This setup keeps your services secure by only exposing them through the Nginx proxy, which handles all SSL termination and access control.