Skip to main content

2 · Collection — Grafana Alloy

DaemonSet on k8s · systemd unit on VMs

GRAFANA ALLOY · ONE BINARY, THREE PIPELINESMetrics pipelineprometheus.scrape → relabel → remote_writeTraces pipelineotelcol.receiver → batch → otelcol.exporterLogs pipelineloki.source.* → loki.process → loki.writeremote_write :8080 /api/v1/pushMimirdistributorOTLP gRPC :4317TempodistributorHTTP :3100 /loki/api/v1/pushLokidistributorstamps X-Scope-OrgID: <tenant> on all pushes — one tenant per team/env

multi-tenancy: Alloy stamps X-Scope-OrgID: <tenant> on all pushes — one tenant per team/env

backendendpoint
Mimirremote_write :8080 /api/v1/push
TempoOTLP gRPC :4317
LokiHTTP :3100 /loki/api/v1/push

Pipelines​

Alloy · metrics pipeline​

Scrapes Prometheus targets, applies relabeling and sends via remote_write to Mimir distributors. Also accepts OTLP metrics.

  • prometheus.scrape with k8s service discovery
  • prometheus.relabel drops high-cardinality junk before it costs money
  • WAL-backed remote_write: retries + backpressure, no data loss on restart
protocolout: remote_write → mimir distributor :8080 /api/v1/push
tenancyX-Scope-OrgID header set per pipeline

Alloy · traces pipeline​

Receives OTLP spans from app SDKs, batches, optionally tail-samples, and exports to Tempo distributors.

  • otelcol.receiver.otlp on :4317 (gRPC) / :4318 (HTTP)
  • otelcol.processor.batch + optional tail_sampling
  • otelcol.exporter.otlp → Tempo
protocolin: OTLP :4317/:4318 · out: OTLP gRPC → tempo distributor
tenancyX-Scope-OrgID header set per pipeline

Alloy · logs pipeline​

Tails container files and journald, parses/relabels, and pushes to Loki distributors.

  • loki.source.kubernetes / .file / .journal
  • loki.process: parse, add labels sparingly (keep cardinality low)
  • loki.write with batching + retries
protocolout: HTTP :3100 /loki/api/v1/push (snappy protobuf)
tenancyX-Scope-OrgID header set per pipeline