Skip to main content

5 · Query & alerting

PromQL · TraceQL · LogQL (via query-frontends)

Grafanadatasources: Mimir · Tempo · Lokiexemplars & trace↔log linksPromQLMimir query-frontendTraceQLTempo query-frontendLogQLLoki query-frontendMimir rulerPromQL rulesLoki rulerLogQL rulesAlertmanagergroup · dedupe · route:9093 /api/v2/alertsfiring alertsNotification targetsPagerDuty · Opsgenie · Slack · email · webhook

Components​

Grafana​

Single pane of glass: dashboards, Explore, and cross-signal correlation across all three backends.

  • Datasources point at the three query-frontends
  • Exemplars: metric spike → trace in one click
  • Derived fields: trace ID in a log line → Tempo trace
  • Trace → logs: Tempo links back to Loki by service labels
protocolPromQL · TraceQL · LogQL over HTTP
tenancydatasource sends X-Scope-OrgID per org/team

Alertmanager​

Receives alerts from Mimir and Loki rulers; groups, dedupes, silences, and routes notifications.

  • Mimir ships a horizontally scalable multi-tenant Alertmanager
  • Routing tree → PagerDuty, Slack, email, webhooks
  • Silences + inhibition rules per tenant
protocolin :9093 /api/v2/alerts
tenancyper-tenant configs + routes

Notification targets​

Where Alertmanager's routing tree delivers alerts. Route by severity/team labels: pages to on-call tools, FYIs to chat.

  • PagerDuty / Opsgenie: critical severity → paging + escalation policies + on-call schedules
  • Slack: warning severity → team channels, no page
  • Email / generic webhook for everything else
  • Route on labels: severity, team, env — set by the rule, not the receiver
protocolPagerDuty Events API v2 · Opsgenie Alert API · Slack webhook