Skip to main content

7 · Multi-IDC / multi-cloud — central Mimir control plane

edge clusters remote_write → one central Mimir · GCP example

Write pathingest · flushRead pathquery · fetchBackground · storagecompaction · rules · bucketsdashed = traffic · animated ON-PREMISE DATA CENTERIDC · edge clusterTarget pods / servicesscraping /metricsGrafana Alloy HAagent mode · WAL · ≥2 replicasInterconnect / VPNAWS · EKS CLUSTERS×N edge clustersTarget pods / servicesscraping /metricsGrafana Alloy HA×2 per cluster · WAL on EBSegress: Transit Gateway → VPN / InterconnectAZURE · AKS CLUSTERS×N edge clustersTarget pods / servicesscraping /metricsGrafana Alloy HA×2 per cluster · WAL on managed diskegress: ExpressRoute / VPNInterconnect / VPNremote_write · JWT/mTLS

Edge sites​

On-premise data center​

Edge/local cluster inside the IDC. Alloy scrapes target pods/services locally and remote_writes to the central Mimir over private connectivity — no telemetry stored on site beyond the WAL buffer.

  • Alloy HA: ≥2 replicas (agent mode, WAL on persistent storage)
  • external_labels: cluster, site, region → identifies origin in queries
  • Link: Dedicated Interconnect or IPsec VPN to the GCP VPC
protocolremote_write HTTPS + JWT/mTLS → central distributor
tenancyX-Scope-OrgID per site or team

AWS edge clusters​

EKS clusters run the same Alloy HA pattern; each cluster pushes to the central control plane rather than running its own Mimir.

  • Alloy ≥2 replicas per cluster · WAL on EBS
  • HA pairs dedupe at the central distributor (HA tracker on cluster/replica labels)
  • Egress via Transit Gateway → VPN/Interconnect to GCP
protocolremote_write HTTPS + JWT/mTLS
tenancytenant per account/env

Azure edge clusters​

AKS clusters — identical pattern. One central query surface means Grafana sees on-prem, AWS and Azure metrics in a single PromQL query.

  • Alloy ≥2 replicas per cluster · WAL on managed disk
  • ExpressRoute / VPN to the GCP VPC
protocolremote_write HTTPS + JWT/mTLS

Alloy HA pattern​

Every edge site runs at least two Alloy replicas scraping the same targets, so a node loss never drops metrics.

  • Both replicas send everything; central HA tracker keeps one copy
  • Labels: cluster=<name>, replica=<a|b> drive dedup
  • WAL survives link outages — replays on reconnect (hours of buffer)
protocolprometheus.remote_write with queue + retry

Target pods / services​

The workloads being monitored at each site. They expose /metrics (or push OTLP) and never talk to the central Mimir directly — only the local Alloy does.

  • Prometheus exporters, app /metrics endpoints, kube-state-metrics
  • Discovered by Alloy via k8s API / static config
  • No egress needed from workloads — Alloy is the only WAN talker
protocolscraped over HTTP inside the cluster

All edge→center traffic is authenticated, encrypted remote_write over private connectivity.

  • HTTPS + JWT bearer token (per tenant) or mTLS client certs
  • Interconnect / ExpressRoute / VPN — no public bucket or ingester exposure
  • Bandwidth ≈ active series × 1-2 B/s after compression — small vs raw logs
protocolremote_write /api/v1/push over Interconnect/VPN
GCP · DEDICATED VPC · GKE — PRIMARY MIMIR CONTROL PLANEWRITE PATHExternal HTTPS LBCloud ArmorprotectedInternal Envoy proxyJWT verifytenant assertDistributorstatelessHA tracker dedupKafka / Strimzibufferregional NVMe SSDIngesterstateful · WALregional PDwrites → validate · tenancy · shard → consume at own pace → blocks cut (2h) → uploadREAD PATHInternal ingressVPC-onlyQuery frontendresults cachequery splittingquery queue · schedulerper-tenantfair queueQuerier ×Npulls jobsmerges recent + hist.Store gatewayhistoricallocal ephemeral SSDreads · Grafana → active data <12h (Ingester)historical · local ephemeral SSD (Store gateway) ← block downloadsGoogle Cloud Storage — source of truthTSDB blocks · Standard → Nearline → ColdlineCompactorGCS merge · dedup · retention

GCP control plane​

External HTTPS LB + Envoy​

Single hardened entry point. Cloud Armor filters at the edge; Envoy terminates auth and routes writes to distributors, reads to the query path.

  • Cloud Armor: WAF + allowlist of site egress IPs
  • Envoy validates JWT/mTLS, injects/asserts X-Scope-OrgID
  • Reads go through internal ingress → query-frontend (Grafana only)
protocolin :443 · out distributor :8080 / query-frontend :8080

Internal Envoy proxy​

Sits behind the external LB on the write path: terminates auth and turns edge identity into tenant identity before traffic touches Mimir.

  • Verifies JWT bearer token or mTLS client cert per site
  • Asserts/injects X-Scope-OrgID — edge can't spoof another tenant
  • Rate-limits misbehaving sites at the front door
protocolin ← HTTPS LB · out → distributor :8080

distributor (central)​

Stateless pool receiving remote_write from every site. The HA tracker dedupes Alloy HA pairs per cluster before anything is stored.

  • Multi-tenant ID check + per-tenant limits at global scale
  • HA tracker: elects one replica per cluster label, drops the twin
  • Autoscales with aggregate ingest of all sites
protocolin /api/v1/push · out → Kafka / ingesters
tenancyper-site/team tenants enforced here

Kafka / Strimzi buffer​

Optional high-throughput buffer between distributors and ingesters: absorbs multi-site ingest spikes and decouples ingester restarts from write availability (Mimir's Kafka-based ingest architecture).

  • Strimzi-operated Kafka on GKE · regional NVMe SSD
  • Ingesters consume at their own pace — backpressure without data loss
  • Replay window covers ingester rollouts/failures
protocolproduce ← distributor · consume → ingester

ingester (central)​

Stateful pool on GKE: regional persistent disks for WAL, zone-aware replication, cuts TSDB blocks and uploads them to GCS.

  • Regional PD = disk survives a zone failure
  • Serves 'active data' to queriers; blocks cut every 2h → GCS
protocolgRPC in · blocks → GCS

compactor (central)​

Merges the per-2h blocks from all ingesters into large deduplicated blocks in GCS and applies retention.

  • Runs on GKE, stateless apart from scratch disk
  • Global dedup of RF copies + HA leftovers
protocolreads/writes GCS

Internal ingress (reads)​

Read path is VPC-internal only: Grafana reaches the query-frontend through an internal ingress, never over the public internet.

  • Grafana datasource → internal ingress → query-frontend
  • Conceptually pairs with the HA tracker note in the original diagram
  • Public LB carries writes only
protocolinternal L7 → query-frontend :8080

query-frontend (central)​

One query surface for the whole company: Grafana asks this endpoint and gets metrics from every IDC and cloud merged.

  • Caching + query splitting + per-tenant queue
  • Internal ingress only — not exposed publicly
protocolPromQL /prometheus/api/v1/*

Query queue / scheduler​

Conceptual queue between frontend and queriers (query-scheduler in Mimir): queriers pull work, giving per-tenant fairness and painless querier autoscaling.

  • Per-tenant fair queuing — one team's heavy dashboard can't starve others
  • Queue depth is the autoscaling signal for queriers
protocolgRPC · queriers pull jobs

querier (central)​

Fans out to ingesters (recent, all sites already merged at write time) and store-gateways (historical blocks from GCS).

  • Cross-site queries are just normal PromQL — data is co-located
  • Scale out for dashboard-heavy orgs
protocolgRPC → ingesters + store-gateways

Ingester (read side)​

Same stateful ingester pool, hit by queriers for active data (~last 12h) that hasn't been compacted into GCS blocks yet.

  • Recent samples answered from the in-memory TSDB head
  • Older ranges come from store-gateways instead
protocolgRPC ← querier

store-gateway (central)​

Serves historical blocks from GCS; local ephemeral SSD caches index headers and hot chunks.

  • Block downloads cached on local SSD → fast repeated queries
  • Zone-aware block sharding
protocolreads GCS bucket

GCS — source of truth​

The only durable store. Every site's metrics end up here as TSDB blocks; everything else can be rebuilt.

  • Lifecycle policies: Standard → Nearline → Coldline as blocks age
  • Cross-region bucket option for DR
  • Equivalent design works with S3 (EKS central) or Azure Blob (AKS central)
protocolGCS API (S3-compatible pattern)