I used to deploy by running commands on my laptop and hoping I typed the right ones. GitOps ends that. The cluster watches Git, Git holds the desired state, and the only way to change production is a merged pull request. This lesson walks the loop from merge to live, hop by hop.
Difficulty: Intermediate · Know this first: Lesson 49, Lesson 33
Every hop charges rent in seconds. The bar below is a worked example for this lesson's scenario - not a measurement from a live system - but the shape is honest: in a GitOps deploy, the waiting usually costs more than the working. Step through the lesson or switch the scenario and watch the bar change.
The budget above is one fixed sync. This box is yours. Move the sliders and watch what happens to the wait between merge and live, how long a hand edit can hide, and how long a full fleet sweep takes. I learned this the slow way: a three minute poll feels free until the fix users are waiting for is sitting in Git, merged, tested, and not live.
How many apps the controller syncs. Each app adds about 15 seconds of apply work to a full fleet sweep, one app at a time. It does not slow down one app's own sync.
How often the controller polls Git. The worst case wait is the full interval, so a change that lands just after a poll waits almost the whole time. The default of 180 seconds is the usual three minute poll.
Try this: (1) Set the sync interval to 10 and the fleet to 10. The lag is about 102 seconds and the drift window is 40 seconds. (2) Set the interval to 600 and the fleet to 50. The lag is about 692 seconds, the drift window is 630 seconds, and the fleet sweep is 1,350 seconds. Same system, same code. Only the knobs changed.
Three questions. No score is saved anywhere - this is just you against the material.
ApplicationSets are the first trade-off. One template generates many Applications. I write the template once, give it a list of apps or clusters, and it creates one ArgoCD Application per entry. Forty apps stop being forty copy-pasted YAML files. The risk moves: a bad template now breaks forty apps at once, so the template gets the same review a production change gets.
Multi-cluster GitOps is the second trade-off. One repo can drive many clusters, or each cluster can have its own repo and its own controller. I prefer one repo with a folder per cluster when the clusters should match, and separate repos when the teams do not share a release train. Either way, a change to shared config must not fan out to every cluster in the same sync. I sync dev first, watch it, then promote the same commit to staging and prod. Promotion is a merge, not a copy.
Sealed secrets are the rule I never bend. Never commit a plaintext secret to Git. Git keeps everything forever, and a secret in history is a secret that has to be rotated. I use SealedSecrets or External Secrets: the value in Git is encrypted and only the cluster can open it, or Git holds a pointer and the real value lives in a vault. If a secret ever lands in Git in plaintext, I rotate it first and clean history second. Rotation is the fix. History cleanup is hygiene.
GitOps for non-Kubernetes targets is the same loop with a bigger blast radius. Terraform plans can run from Git the same way, with a controller that applies the plan when the repo changes. The same rules hold: Git is the truth, the apply is a pull, and a bad plan fails before it touches anything. What changes is what a mistake can do. A bad Terraform apply can delete a database, so non-Kubernetes targets get manual approval on the sync, at least for prod.
Sync windows and approval are how I keep prod boring. Prod gets a sync window: no auto-sync during the change freeze, and no auto-sync at all for the riskiest apps. For those, the controller computes the diff and waits for a human to press sync. That is still GitOps. The decision is recorded in Git, and the apply is still done by the controller, not by a laptop.
At 10x, forty apps become four hundred, and one controller polling one repo every three minutes starts to lag. I split repos by team, add webhooks so polls stop being the wait, and watch the controller's own CPU and memory. The controller is now a production service, and it gets the same alerts as everything else.
At 100x, many clusters and many teams are one bad template away from a fleet-wide incident. I add policy checks in CI that reject manifests before they merge: no latest tags, no missing resource limits, no plaintext secrets. The repo gets a CODEOWNERS file so a team cannot change another team's folder. And I rehearse the worst day: the Git host is down. The clusters keep running on the last synced state, and I can still roll back by pointing the controller at a mirror.
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: checkout
namespace: argocd
spec:
project: default
source:
repoURL: https://git.example.com/team/checkout.git
targetRevision: main
path: k8s/overlays/prod
destination:
server: https://kubernetes.default.svc
namespace: checkout
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
What the snippet means: the Application says where the truth lives and where it goes. The repo, the branch, and the folder are the source. The cluster and namespace are the destination. Automated sync with prune means the controller applies changes on its own and deletes objects that Git no longer mentions. Self-heal means a hand edit gets reverted. CreateNamespace means the controller makes the namespace if it is missing. Those six lines are the whole contract. Everything else in this lesson is what happens when that contract runs.
During a traffic spike, the on-call engineer scaled a Deployment from three replicas to eight by hand, with kubectl, to shed load. Three minutes later ArgoCD self-heal put it back to three. Latency spiked again. The engineer scaled it again. The controller reverted it again. They fought the loop for twenty minutes while users waited.
Three things. First, the runbook now says: during an incident, suspend the Application in ArgoCD before you touch anything live, then edit, then commit the change to Git before you re-enable sync. Second, a suspended Application pages after one hour, so a paused loop cannot be forgotten overnight. Third, the dashboard got a panel listing every suspended app, so the next on-call can see at a glance which loops are off. The lesson is not that self-heal is bad. The lesson is that a control loop and a human fixing the same thing at the same time will fight, and the human has to pause the loop first.
You run 40 apps across 3 clusters: dev, staging, and prod. Each app has its own repo today and deploys through a pipeline that holds cluster credentials. Design the GitOps setup. How do the repos look, where do secrets live, how does a change reach prod, and what stops a bad manifest from taking down all three clusters?
Start with the numbers from this lesson. One app syncs in about 15 seconds of apply work after the poll. Forty apps, synced one at a time after a 180 second poll, take 180 + 40 x 15 = 780 seconds, about 13 minutes for a full fleet sweep. That is the worst case, and it is too slow for a hot fix, so I add a webhook: the lag for one app drops to the fixed 92 seconds, and the fleet sweep is still bounded by the 15 second apply per app. I put each app in one repo per team, not one repo per app, so forty apps become five or six repos. Each cluster gets a folder per environment in the same repo, and promotion from staging to prod is a merge of one commit, not a fresh deploy.
One repo per team buys fewer repos to watch and costs a bigger blast radius if shared config breaks. Separate repos buy isolation and cost forty controllers to feed. Auto-sync in dev and staging buys speed and costs the occasional surprise. Manual sync in prod buys a human in the loop and costs a few minutes on every deploy. Sealed secrets buy Git as the single source of truth and cost a rotation plan for the day a secret leaks. And the pipeline loses its cluster credentials, which is a security win, but the team has to learn to debug through the ArgoCD UI instead of kubectl, and that is a real learning curve to name out loud.