A canary that passes every check on the eu-west cluster says nothing about what happens on us-east: different network latency to external dependencies, different load, sometimes a few Kubernetes minor versions apart. Treating a multi-cluster rollout as a single-cluster rollout repeated in parallel removes exactly the safeguard the canary was supposed to provide.

Sequential rollout between clusters, not just between pods

A classic rolling deployment sequences pod replacement inside one cluster. At multi-cluster scale, the same logic applies one level up: deploy first to a low-impact cluster (internal traffic, a low-volume region), observe, then progressively extend to the next clusters in waves, never all at once. The generation mechanism is the same ApplicationSet detailed in the article on GitOps at scale; here, what matters is the propagation order between waves, not the generation itself.

# ArgoCD ApplicationSet: wave rollout via a list generator combined
# with a progressive rollout strategy
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: payment-api
spec:
  generators:
    - list:
        elements:
          - cluster: canary-eu-west
            wave: "1"
          - cluster: prod-eu-west
            wave: "2"
          - cluster: prod-us-east
            wave: "3"
          - cluster: prod-ap-south
            wave: "3"
  template:
    metadata:
      annotations:
        argocd.argoproj.io/sync-wave: "{{wave}}"
    spec:
      destination:
        server: "{{cluster}}"

Clusters in the same wave deploy in parallel with each other; a wave only advances to the next once the preceding one is healthy. This structure turns a manual checklist (“did we verify eu-west before touching us-east?”) into a constraint enforced by the deployment system itself.

What decides a wave is healthy

A wave that advances automatically with no explicit health criterion is just a disguised parallel rollout. The criterion has to be measurable and automated, typically an error rate or a latency percentile compared against a reference window, not just the presence of Ready pods:

# Argo Rollouts analysis, referenced by each wave before promotion
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  name: error-rate-check
spec:
  metrics:
    - name: error-rate
      interval: 2m
      successCondition: result[0] < 0.01
      provider:
        prometheus:
          address: http://prometheus.monitoring:9090
          query: |
            sum(rate(http_requests_total{status=~"5..",cluster="{{args.cluster}}"}[5m]))
            / sum(rate(http_requests_total{cluster="{{args.cluster}}"}[5m]))

Without this automated guardrail, propagation speed between clusters becomes the only available safety lever: slowing down manually while hoping to eyeball a problem, which doesn’t hold up at the scale of an organization running more than two or three clusters.

Rollback follows the same topology

A poorly designed multi-cluster rollback reverts everywhere at once, which cancels the one advantage wave-based rollout bought in the first place: limiting blast radius. The rule symmetric to wave-based rollout applies to rollback too: revert only the clusters where analysis failed, leave already-validated clusters in place. A pipeline that doesn’t distinguish this case treats every anomaly as a global incident, even when only one cluster out of five is actually affected.

What network latency changes about the analysis

The same error threshold applied uniformly across every cluster ignores an operational reality: a cluster remote from a critical external dependency (a centralized database, for instance) has structurally higher latency than the cluster hosting it. A single threshold for all clusters forces either a threshold too loose for the nearby cluster, or one too strict that triggers spurious rollbacks on the remote one. The threshold needs calibrating per cluster, against its own historical reference window, never against one global value.

Takeaway

A multi-cluster canary is only a guarantee if the rollout progresses in measured waves, with an automated, per-cluster-calibrated health criterion, never in parallel across the whole fleet. Rollback needs to follow the same granularity as rollout: target the failing clusters, leave the rest alone. Same discipline as progressive delivery from CI, extended to the level where it actually protects an infrastructure spread across multiple regions.