Grafana Dashboards

Data Sources and Dashboard Anatomy

Grafana renders metrics; it computes almost nothing itself. Point it at Prometheus through a provisioned data source so environments do not drift:

apiVersion: 1
datasources:
- name: Prometheus
  type: prometheus
  url: http://prometheus-server.observability:9090
  isDefault: true

A dashboard is panels over queries: time series for trends, stat/gauge for current values, table for breakdowns. Panels should map to golden signals — traffic (rate(requests_total[5m])), errors (5xx ratio), latency (p95/p99), and saturation (requests vs allocatable).

Variables and Provisioned Dashboards

Template variables make one dashboard serve every team:

Variable: namespace   query = label_values(kube_pod_info, namespace)
Variable: workload    query = label_values(kube_pod_info{namespace="$namespace"}, pod)
Panel:    sum by (pod) (rate(http_requests_total{job=~"$workload"}[5m]))

Store dashboard JSON in Git and ship it with a file provider so panels are reviewable and versioned:

apiVersion: 1
providers:
- name: platform
  folder: SRE
  options: { path: /var/lib/grafana/dashboards }

Alerting on symptoms belongs in Prometheus rules with Alertmanager as the single paging path; Grafana alerting can supplement it for board-only notices.

Pitfalls and Interview Angles

Unbounded variables. A variable fed by a high-cardinality label floods the dropdown and the query — restrict with regex or record-rule labels.

Raw bucket queries in every panel. Computing quantiles on the fly across long ranges burns query time; pre-aggregate with recording rules (e.g. job:http_request_duration_seconds:p95) and point panels there.

Redundant sums. sum(rate(...)) grouped one way on each panel silently double-counts when combining panels — keep aggregations consistent.

Interview angle: describe how you would build an SLO dashboard (ratio panel from two counters, latency from histogram_quantile), and say why dashboard JSON belongs in version control.