Grafana Complete Guide: Dashboards and Observability for DevOps

What is Grafana?

Grafana is an open-source observability and data visualization platform created by Grafana Labs. It allows you to query, visualize, alert on, and understand metrics, logs, and traces from any storage system. Grafana connects to over 100 data sources out of the box, including Prometheus, Graphite, InfluxDB, Elasticsearch, PostgreSQL, MySQL, and cloud services like AWS CloudWatch and Azure Monitor.

Originally focused on time-series visualization for infrastructure monitoring, Grafana has evolved into a full observability platform with the LGTM stack: Loki (logs), Grafana (visualization), Tempo (traces), and Mimir (metrics). DevOps teams, SREs, and platform engineers use Grafana to monitor everything from Kubernetes clusters to application performance to business KPIs.

Installing Grafana

Grafana can be installed via multiple methods depending on your environment. Below are the most common approaches.

Docker (Recommended for Development)

# Run Grafana OSS with Docker
docker run -d \
  --name grafana \
  -p 3000:3000 \
  -v grafana-storage:/var/lib/grafana \
  -e "GF_SECURITY_ADMIN_PASSWORD=strongpassword" \
  grafana/grafana-oss:latest

# Docker Compose with Prometheus + Grafana
# docker-compose.yml
version: "3.9"
services:
  prometheus:
    image: prom/prometheus:latest
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml
    ports:
      - "9090:9090"

  grafana:
    image: grafana/grafana-oss:latest
    depends_on:
      - prometheus
    ports:
      - "3000:3000"
    volumes:
      - grafana-data:/var/lib/grafana
    environment:
      - GF_SECURITY_ADMIN_PASSWORD=admin

volumes:
  grafana-data:

APT (Ubuntu / Debian)

# Install prerequisites
sudo apt-get install -y apt-transport-https software-properties-common wget

# Add Grafana GPG key and repository
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | \
  gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null
echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] \
  https://apt.grafana.com stable main" | \
  sudo tee /etc/apt/sources.list.d/grafana.list

# Install and start Grafana
sudo apt-get update
sudo apt-get install grafana
sudo systemctl enable grafana-server
sudo systemctl start grafana-server

# Check status
sudo systemctl status grafana-server

Helm (Kubernetes)

# Add the Grafana Helm repository
helm repo add grafana https://grafana.github.io/helm-charts
helm repo update

# Install Grafana with custom values
helm install grafana grafana/grafana \
  --namespace monitoring \
  --create-namespace \
  --set persistence.enabled=true \
  --set persistence.size=10Gi \
  --set adminPassword=strongpassword \
  --set service.type=LoadBalancer

# Get the admin password if auto-generated
kubectl get secret --namespace monitoring grafana \
  -o jsonpath="{.data.admin-password}" | base64 --decode

Connecting Data Sources

Data sources are the backbone of Grafana. Each data source is a storage backend that Grafana queries to fetch metrics, logs, or traces. You can configure data sources via the Grafana UI or provision them as code.

Prometheus

Prometheus is the most common data source for Grafana. It uses PromQL (Prometheus Query Language) for querying time-series metrics. Prometheus scrapes targets at regular intervals, storing data in its own time-series database.

# prometheus.yml - Scrape configuration
global:
  scrape_interval: 15s
  evaluation_interval: 15s

scrape_configs:
  - job_name: "prometheus"
    static_configs:
      - targets: ["localhost:9090"]

  - job_name: "node-exporter"
    static_configs:
      - targets: ["node-exporter:9100"]

  - job_name: "application"
    metrics_path: /metrics
    static_configs:
      - targets: ["app:8080"]

# Common PromQL queries for Grafana dashboards:
# CPU usage: rate(node_cpu_seconds_total{mode!="idle"}[5m])
# Memory: node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes
# HTTP rate: rate(http_requests_total[5m])
# Error rate: rate(http_requests_total{status=~"5.."}[5m])

Loki (Logs)

# Grafana data source configuration for Loki
# Configuration > Data Sources > Add > Loki
# URL: http://loki:3100

# LogQL query examples in Grafana:
# All logs from a specific app:
#   {app="frontend"}

# Filter logs containing "error":
#   {app="frontend"} |= "error"

# Regex filter:
#   {app="api"} |~ "status=(4|5)\\d{2}"

# Parse JSON logs and filter:
#   {app="api"} | json | level="error" | line_format "{{.message}}"

InfluxDB

# InfluxDB 2.x data source configuration
# URL: http://influxdb:8086
# Organization: my-org
# Token: <your-influxdb-token>
# Default Bucket: my-bucket

# Flux query example:
from(bucket: "my-bucket")
  |> range(start: -1h)
  |> filter(fn: (r) => r._measurement == "cpu")
  |> filter(fn: (r) => r._field == "usage_user")
  |> aggregateWindow(every: 1m, fn: mean)
  |> yield(name: "mean")

Elasticsearch

# Elasticsearch data source configuration
# URL: http://elasticsearch:9200
# Index name: logs-*
# Time field: @timestamp
# Version: 8.x

# In Grafana, use the query editor to build
# Lucene or KQL queries:
#   level:error AND service:api
#   status:>=400 AND method:POST

# Aggregations: Date Histogram, Terms,
# Percentiles, Extended Stats, etc.

Creating Dashboards

Dashboards are the core of Grafana. A dashboard is a collection of panels arranged on a grid, each visualizing data from one or more queries. You can create dashboards manually via the UI or import community dashboards from grafana.com/grafana/dashboards.

To create a new dashboard, click the + icon in the sidebar and select New dashboard. Then click Add visualization to add your first panel. Select a data source, write your query, choose a panel type, and configure display options.

Dashboard JSON Model

{
  "dashboard": {
    "title": "Application Overview",
    "tags": ["production", "api"],
    "timezone": "browser",
    "refresh": "30s",
    "time": {
      "from": "now-6h",
      "to": "now"
    },
    "panels": [
      {
        "type": "timeseries",
        "title": "Request Rate",
        "gridPos": { "h": 8, "w": 12, "x": 0, "y": 0 },
        "targets": [
          {
            "expr": "rate(http_requests_total[5m])",
            "legendFormat": "{{method}} {{path}}"
          }
        ]
      },
      {
        "type": "stat",
        "title": "Total Requests (24h)",
        "gridPos": { "h": 4, "w": 6, "x": 12, "y": 0 },
        "targets": [
          {
            "expr": "increase(http_requests_total[24h])"
          }
        ]
      }
    ]
  }
}

Panel Types

Grafana offers a rich library of panel types. Choosing the right panel is critical for effective data visualization.

Time Series

The default and most common panel type. Ideal for displaying metrics over time with support for line, area, bar, and point visualizations. Supports multiple queries, legend formatting, axis configuration, thresholds, and overrides.

# PromQL queries for a Time Series panel

# Request rate by status code
sum by (status_code) (
  rate(http_requests_total[5m])
)

# 95th percentile latency
histogram_quantile(0.95,
  sum by (le) (
    rate(http_request_duration_seconds_bucket[5m])
  )
)

Stat Panel

Displays a single numeric value with optional sparkline. Use it for high-level KPIs like total requests, error count, or uptime percentage. Supports color thresholds and value mappings.

Gauge Panel

Shows a value within a range, perfect for CPU usage, memory utilization, or disk space. Configure min/max values and threshold colors (green, yellow, red) to quickly identify problems.

Table Panel

Renders data in a tabular format. Useful for displaying instant query results, top-N lists, or detailed breakdowns. Supports column filtering, sorting, cell coloring, and links.

Heatmap Panel

Visualizes data density over two dimensions. Ideal for histogram distributions (like request latency buckets) and understanding patterns across time. Works well with Prometheus histogram metrics.

Bar Chart Panel

Compares categorical data with horizontal or vertical bars. Use it for top-N endpoints by request count, error rates by service, or resource usage comparison across teams.

Variables and Templating

Variables make dashboards dynamic and reusable. Instead of hardcoding values like server names or environments, you define variables that appear as dropdown menus at the top of the dashboard. Users can switch between environments, namespaces, or services without editing queries.

# Variable configuration examples

# Query variable - fetches values from Prometheus
# Name: namespace
# Type: Query
# Query: label_values(kube_pod_info, namespace)
# Multi-value: enabled
# Include All: enabled

# Using variables in panel queries:
rate(http_requests_total{namespace=~"\$namespace"}[5m])

# Custom variable
# Name: environment
# Type: Custom
# Values: production, staging, development

# Interval variable
# Name: interval
# Type: Interval
# Values: 1m, 5m, 15m, 1h
# Used in: rate(http_requests_total[\$interval])

# Chained variables - filter pods by selected namespace
# Name: pod
# Query: label_values(kube_pod_info{namespace=~"\$namespace"}, pod)

Alerting Rules and Notifications

Grafana Alerting (unified alerting since Grafana 9) lets you define alert rules that evaluate queries on a schedule. When conditions are met, notifications are sent to configured contact points. The alerting system supports multi-dimensional alerts, routing, silences, and mute timings.

Creating Alert Rules

# Alert rule configuration (YAML provisioning)
# File: provisioning/alerting/rules.yml
apiVersion: 1
groups:
  - orgId: 1
    name: application-alerts
    folder: Production
    interval: 1m
    rules:
      - uid: high-error-rate
        title: High Error Rate
        condition: C
        data:
          - refId: A
            relativeTimeRange:
              from: 300
              to: 0
            datasourceUid: prometheus
            model:
              expr: >
                sum(rate(http_requests_total{status=~"5.."}[5m]))
                /
                sum(rate(http_requests_total[5m]))
          - refId: B
            datasourceUid: __expr__
            model:
              type: reduce
              reducer: last
              expression: A
          - refId: C
            datasourceUid: __expr__
            model:
              type: threshold
              expression: B
              conditions:
                - evaluator:
                    type: gt
                    params: [0.05]
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: Error rate above 5%

Contact Points and Notification Policies

# provisioning/alerting/contactpoints.yml
apiVersion: 1
contactPoints:
  - orgId: 1
    name: slack-ops
    receivers:
      - uid: slack-1
        type: slack
        settings:
          recipient: "#ops-alerts"
          token: xoxb-your-slack-token
          title: '{{ template "slack.default.title" . }}'
  - orgId: 1
    name: pagerduty-critical
    receivers:
      - uid: pd-1
        type: pagerduty
        settings:
          integrationKey: your-pd-integration-key
          severity: critical

# provisioning/alerting/policies.yml
apiVersion: 1
policies:
  - orgId: 1
    receiver: slack-ops
    group_by: [alertname, namespace]
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h
    routes:
      - receiver: pagerduty-critical
        matchers:
          - severity = critical
        continue: false

Provisioning: Dashboards and Data Sources as Code

Provisioning allows you to configure Grafana from files instead of the UI. This is essential for infrastructure-as-code workflows, version-controlled configurations, and automated deployments. Provisioning files are placed in Grafana's provisioning/ directory.

Data Source Provisioning

# provisioning/datasources/datasources.yml
apiVersion: 1
datasources:
  - name: Prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false

  - name: Loki
    type: loki
    access: proxy
    url: http://loki:3100
    jsonData:
      derivedFields:
        - name: TraceID
          matcherRegex: "traceID=(\\w+)"
          url: "\$\${__value.raw}"
          datasourceUid: tempo

  - name: Tempo
    type: tempo
    access: proxy
    url: http://tempo:3200
    uid: tempo
    jsonData:
      tracesToLogsV2:
        datasourceUid: loki
        filterByTraceID: true

Dashboard Provisioning

# provisioning/dashboards/dashboards.yml
apiVersion: 1
providers:
  - name: "default"
    orgId: 1
    folder: "Provisioned"
    type: file
    disableDeletion: true
    updateIntervalSeconds: 30
    allowUiUpdates: false
    options:
      path: /var/lib/grafana/dashboards
      foldersFromFilesStructure: true

# Place dashboard JSON files in /var/lib/grafana/dashboards/
# Grafana will auto-load them on startup
# Directory structure becomes folder structure in Grafana:
#   dashboards/
#     infrastructure/
#       node-exporter.json
#       kubernetes.json
#     application/
#       api-overview.json
#       frontend-metrics.json

Grafana Loki: Log Aggregation

Grafana Loki is a horizontally scalable, highly available log aggregation system inspired by Prometheus. Unlike Elasticsearch, Loki indexes only the labels (metadata) of logs rather than the full log content, making it significantly cheaper to operate while still being fast for label-based queries.

# docker-compose for Loki + Promtail + Grafana
services:
  loki:
    image: grafana/loki:latest
    ports:
      - "3100:3100"
    volumes:
      - ./loki-config.yml:/etc/loki/config.yml
    command: -config.file=/etc/loki/config.yml

  promtail:
    image: grafana/promtail:latest
    volumes:
      - /var/log:/var/log
      - ./promtail-config.yml:/etc/promtail/config.yml
    command: -config.file=/etc/promtail/config.yml

# promtail-config.yml
server:
  http_listen_port: 9080

positions:
  filename: /tmp/positions.yaml

clients:
  - url: http://loki:3100/loki/api/v1/push

scrape_configs:
  - job_name: system
    static_configs:
      - targets: [localhost]
        labels:
          job: varlogs
          __path__: /var/log/*.log

LogQL Query Examples

# Stream selector - select logs by labels
{job="api", namespace="production"}

# Line filter - case sensitive contains
{job="api"} |= "error"

# Negative filter - exclude lines
{job="api"} != "healthcheck"

# Regex filter
{job="api"} |~ "status=(4|5)\\d{2}"

# JSON parser - extract fields from JSON logs
{job="api"} | json | level="error" | duration > 1s

# Log metrics - count errors per minute
sum(rate({job="api"} |= "error" [1m])) by (service)

# Unwrap numeric values from logs
avg_over_time(
  {job="api"} | json | unwrap duration [5m]
) by (endpoint)

Grafana Tempo: Distributed Tracing

Grafana Tempo is a distributed tracing backend that requires only object storage (S3, GCS, Azure Blob) to operate. It accepts traces in Jaeger, Zipkin, OpenTelemetry, and OpenCensus formats. Tempo integrates tightly with Grafana, Loki, and Mimir to correlate traces with logs and metrics.

# tempo-config.yml
server:
  http_listen_port: 3200

distributor:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: 0.0.0.0:4317
        http:
          endpoint: 0.0.0.0:4318
    jaeger:
      protocols:
        thrift_http:
          endpoint: 0.0.0.0:14268

storage:
  trace:
    backend: s3
    s3:
      bucket: tempo-traces
      endpoint: s3.amazonaws.com
      region: us-east-1

metrics_generator:
  registry:
    external_labels:
      source: tempo
  storage:
    path: /var/tempo/generator/wal
    remote_write:
      - url: http://prometheus:9090/api/v1/write

Grafana Cloud

Grafana Cloud is the fully managed SaaS platform from Grafana Labs. It includes hosted Grafana, Prometheus (Mimir), Loki, Tempo, and additional features like synthetic monitoring, k6 load testing, and Incident Response Management (IRM). The free tier includes 10,000 series for metrics, 50 GB of logs, and 50 GB of traces per month.

# Sending metrics to Grafana Cloud Prometheus
# Add remote_write to your prometheus.yml:
remote_write:
  - url: https://prometheus-prod-01-eu-west-0.grafana.net/api/prom/push
    basic_auth:
      username: <instance_id>
      password: <api_key>

# Sending logs to Grafana Cloud Loki
# Update your Promtail or Alloy config:
clients:
  - url: https://logs-prod-eu-west-0.grafana.net/loki/api/v1/push
    basic_auth:
      username: <instance_id>
      password: <api_key>

# Grafana Alloy (the OpenTelemetry collector by Grafana)
# is the recommended agent for Grafana Cloud
# It replaces Promtail, Grafana Agent, and OTel Collector

User Management and RBAC

Grafana supports Organizations, Teams, and Role-Based Access Control (RBAC) for fine-grained permission management. RBAC is available in Grafana Enterprise and Grafana Cloud.

• Organizations: Isolated tenants with separate dashboards, data sources, and users. Each user can belong to multiple organizations.
• Teams: Groups of users within an organization. Assign dashboard and folder permissions to teams.
• Roles: Admin, Editor, Viewer. Custom roles (Enterprise/Cloud) allow granular permissions like data source access, alert rule management, or annotation creation.

# grafana.ini - Authentication configuration
[auth]
disable_login_form = false

# LDAP authentication
[auth.ldap]
enabled = true
config_file = /etc/grafana/ldap.toml

# OAuth2 / OIDC (e.g., Keycloak, Okta, Azure AD)
[auth.generic_oauth]
enabled = true
name = SSO
client_id = grafana
client_secret = your-client-secret
scopes = openid profile email
auth_url = https://sso.example.com/auth
token_url = https://sso.example.com/token
api_url = https://sso.example.com/userinfo
role_attribute_path = role

# Auto-assign roles based on org
[users]
auto_assign_org = true
auto_assign_org_role = Viewer

Embedding and Sharing Dashboards

Grafana provides multiple ways to share dashboards and panels with stakeholders. You can share via direct links, snapshots, embedded iframes, or public dashboards.

• Direct link: Share the dashboard URL with time range and variable parameters preserved
• Snapshot: Create a point-in-time snapshot that does not require authentication to view
• Embedded panels: Embed individual panels into external web pages using iframe URLs
• Public dashboards (Grafana 10+): Make a dashboard publicly accessible without login
• PDF/PNG reports (Enterprise): Schedule automated reports delivered via email

# Enable public dashboards and anonymous access
# grafana.ini
[feature_toggles]
publicDashboards = true

[auth.anonymous]
enabled = true
org_name = Public
org_role = Viewer

# Embedding: allow Grafana to be loaded in iframes
[security]
allow_embedding = true
cookie_samesite = none
cookie_secure = true

# Embed URL pattern:
# https://grafana.example.com/d-solo/DASHBOARD_UID/
#   ?orgId=1&panelId=2&from=now-6h&to=now
#
# Use in HTML:
# <iframe src="..." width="100%" height="400"
#   frameborder="0"></iframe>

Grafana Plugins

Grafana's plugin system extends its functionality with additional data sources, panel types, and applications. Plugins are available on the official Grafana plugin catalog at grafana.com/grafana/plugins.

• Data source plugins: Connect to additional backends (Datadog, Splunk, MongoDB, GitHub, Jira, Snowflake)
• Panel plugins: New visualization types (Flowcharting, Worldmap, Treemap, Diagram, Polystat)
• App plugins: Complete applications like Kubernetes monitoring, synthetic monitoring, and performance testing

# Install plugins via CLI
grafana-cli plugins install grafana-piechart-panel
grafana-cli plugins install grafana-worldmap-panel
grafana-cli plugins install grafana-clock-panel

# Install via Docker environment variable
docker run -d \
  -e "GF_INSTALL_PLUGINS=grafana-clock-panel,grafana-piechart-panel" \
  grafana/grafana-oss:latest

# Install via Helm values
# values.yaml
plugins:
  - grafana-piechart-panel
  - grafana-worldmap-panel
  - grafana-clock-panel

# List installed plugins
grafana-cli plugins ls

# Update all plugins
grafana-cli plugins update-all

Grafana Best Practices for Production

Dashboard Organization

• Use folders to organize dashboards by team, service, or environment
• Follow a naming convention: [Team] Service - View (e.g., Platform - Kubernetes Overview)
• Create a home dashboard with links to the most important dashboards
• Use dashboard tags for easy filtering and search
• Limit dashboards to 15-20 panels for performance

Query Performance

• Use recording rules in Prometheus to precompute expensive queries
• Set appropriate time ranges; avoid querying more data than needed
• Use the Min interval setting to prevent over-sampling on wide time ranges
• Enable query caching to reduce load on data sources

# Prometheus recording rules for Grafana performance
# rules.yml
groups:
  - name: http_metrics
    interval: 1m
    rules:
      # Pre-compute request rate
      - record: job:http_requests:rate5m
        expr: sum by (job) (rate(http_requests_total[5m]))

      # Pre-compute error rate percentage
      - record: job:http_errors:ratio5m
        expr: >
          sum by (job) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (job) (rate(http_requests_total[5m]))

      # Pre-compute p95 latency
      - record: job:http_duration:p95_5m
        expr: >
          histogram_quantile(0.95,
            sum by (job, le) (
              rate(http_request_duration_seconds_bucket[5m])
            )
          )

High Availability

• Run multiple Grafana instances behind a load balancer
• Use a shared external database (PostgreSQL or MySQL) instead of the default SQLite
• Configure shared session storage with Redis or Memcached
• Use Grafana Image Renderer as a separate service for rendering panels to PNG

# grafana.ini - High Availability configuration
[database]
type = postgres
host = pg-primary:5432
name = grafana
user = grafana
password = secret
ssl_mode = require

[remote_cache]
type = redis
connstr = addr=redis:6379,pool_size=100,db=0

[unified_alerting]
enabled = true
ha_listen_address = "0.0.0.0:9094"
ha_peers = "grafana-1:9094,grafana-2:9094,grafana-3:9094"

[rendering]
server_url = http://grafana-image-renderer:8081/render
callback_url = http://grafana:3000/

Security Best Practices

• Change the default admin password immediately after installation
• Use HTTPS with valid TLS certificates (terminate at load balancer or configure in Grafana)
• Enable SSO/OAuth and disable basic authentication in production
• Set cookie_secure = true and cookie_samesite = strict
• Use API keys or service accounts with least-privilege roles for automation
• Restrict data source access to prevent users from running arbitrary queries
• Regularly update Grafana to patch security vulnerabilities

Grafana HTTP API

Grafana exposes a comprehensive HTTP API for automation. You can manage dashboards, data sources, users, organizations, annotations, and alerts programmatically.

# Create a service account and token
curl -X POST http://localhost:3000/api/serviceaccounts \
  -H "Authorization: Bearer admin-token" \
  -H "Content-Type: application/json" \
  -d '{"name": "ci-deploy", "role": "Editor"}'

# Search dashboards
curl -H "Authorization: Bearer sa-token" \
  "http://localhost:3000/api/search?query=kubernetes"

# Export a dashboard
curl -H "Authorization: Bearer sa-token" \
  "http://localhost:3000/api/dashboards/uid/abc123"

# Import a dashboard
curl -X POST http://localhost:3000/api/dashboards/db \
  -H "Authorization: Bearer sa-token" \
  -H "Content-Type: application/json" \
  -d @dashboard.json

# Create an annotation
curl -X POST http://localhost:3000/api/annotations \
  -H "Authorization: Bearer sa-token" \
  -H "Content-Type: application/json" \
  -d '{"dashboardId":1,"text":"Deployment v2.3.1","tags":["deploy"]}'

Frequently Asked Questions

Conclusion

Grafana has become the industry standard for observability and monitoring dashboards. Whether you are monitoring a single application or a fleet of Kubernetes clusters, Grafana provides the visualization, alerting, and collaboration features you need. Start with Docker for local development, connect Prometheus for metrics and Loki for logs, and gradually adopt provisioning as code and Grafana Cloud as your observability needs grow.

The combination of Grafana, Prometheus, Loki, and Tempo (the LGTM stack) provides a complete, open-source observability solution that scales from small teams to enterprise deployments. By following the best practices in this guide, you will build maintainable, performant dashboards that give your team real-time visibility into your systems.