GitOps with ArgoCD

The 2 AM Diff

#

Picture this. Something’s wrong in production. You check the Kubernetes manifests in Git — they look fine. You check what’s actually running in the cluster — it’s different. Someone ran kubectl apply directly three weeks ago, didn’t commit the change, and now nobody remembers what was modified or why. You’re now debugging a mystery diff at 2 AM.

This is the problem GitOps solves. And ArgoCD is how we solved it.

Git Is the Boss Now

#

GitOps is a simple idea: Git is the single source of truth for your cluster state. What’s in the repo is what should be running. If the cluster drifts from Git, something is wrong and needs to be fixed — automatically.

The old way:

Developer → CI builds image → CI runs kubectl apply → cluster updated (maybe)

The GitOps way:

Developer → CI builds image → CI updates manifest in Git → ArgoCD syncs cluster

Nobody runs kubectl apply anymore. Nobody needs cluster credentials to deploy. You push to Git, ArgoCD handles the rest.

One App, One Repo

#

Each service has its own Git repo with a k8s/ directory containing Kustomize manifests (deployments, services, HPAs). A separate infrastructure repo holds ArgoCD Application definitions — one per service:

infra-repo/
├── applications/
│   ├── api-service.yaml        # points to api-service Git repo
│   ├── scheduler-service.yaml  # points to scheduler Git repo
│   └── worker-service.yaml     # points to worker Git repo

Each ArgoCD Application tells ArgoCD: “watch this repo, this path, deploy to this namespace.”

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: api-service
  namespace: argocd
spec:
  source:
    repoURL: https://github.com/my-org/api-service.git
    targetRevision: HEAD
    path: k8s                       # directory with Kustomize manifests
  destination:
    server: https://kubernetes.default.svc
    namespace: production
  syncPolicy:
    automated:
      prune: true                   # delete resources removed from Git
      selfHeal: true                # revert manual cluster changes
    syncOptions:
      - CreateNamespace=true
    retry:
      limit: 5                     # retry failed syncs
      backoff:
        duration: 5s               # start with 5s delay
        factor: 2                  # double each retry
        maxDuration: 3m            # cap at 3 minutes

The Four Sync Superpowers

#

Auto-sync: push to Git, cluster updates automatically. No buttons to click, no pipelines to trigger. ArgoCD polls the repo (or receives a webhook) and applies the changes.

Prune: when you delete a resource from Git, ArgoCD deletes it from the cluster. Without prune, you end up with orphaned deployments, services, and config maps that nobody remembers creating.

Self-heal: if someone runs kubectl edit and modifies a deployment manually, ArgoCD detects the drift and reverts it. Git is the truth. Always.

Retry with backoff: if a sync fails (transient network issue, API server overloaded), ArgoCD retries with exponential backoff — 5s, 10s, 20s, 40s, up to 3 minutes. Most transient failures resolve themselves within a few retries.

The Image Updater

#

Here’s the flow that ties CI/CD to GitOps:

1. Developer pushes code to the service repo
2. CI builds a new Docker image, tags it v1.3.0, pushes to the container registry
3. ArgoCD Image Updater watches the registry
4. It sees v1.3.0, checks the semver constraint, updates the Application
5. ArgoCD syncs the cluster with the new image

The semver filtering is important. We tag QA builds as qa-v1.0.x and production releases as v1.3.0. The Image Updater is configured to only match proper semver tags:

annotations:
  argocd-image-updater.argoproj.io/image-list: app=registry.example.com/my-service
  argocd-image-updater.argoproj.io/app.update-strategy: semver
  argocd-image-updater.argoproj.io/app.allow-tags: "regexp:^v[0-9]+\.[0-9]+\.[0-9]+$"

This regex (^v[0-9]+\.[0-9]+\.[0-9]+$) only matches vX.Y.Z — no qa- prefixes, no latest, no dev- tags. Production deployments only happen when someone explicitly tags a release.

QA vs Production

#

The two environments have different trigger mechanisms:

QA deploys on every push to develop. It’s fast, it’s loose, and if something breaks, nobody’s losing sleep over it. Production only deploys when a semver tag is pushed — an intentional, auditable action.

The Cultural Shift

#

The hardest part of adopting ArgoCD wasn’t the technology. It was getting everyone to stop running kubectl. The first few weeks, someone would inevitably kubectl apply a quick fix directly to the cluster. ArgoCD would detect the drift, revert it, and the fix would disappear. Frustrating at first, but it enforced the right behavior: if it’s not in Git, it doesn’t exist.

Now, deploying is a pull request. Rollback is a revert. The cluster state is auditable through Git history. And nobody debugs mystery diffs at 2 AM anymore (mostly).

The cluster deploys itself now. But how do we know if it’s healthy? Monitoring is next .