From Commit to Cluster: End-to-End Operator Release Pipeline

Releasing a Kubernetes Operator is not the same as deploying a stateless application. The release includes a controller image, CRDs, RBAC, webhooks, manifests or Helm charts, optional OLM bundles, and a rollback story that must respect Kubernetes API compatibility.

Most people searching for a Kubernetes Operator release pipeline want a production path from Git tag to running cluster:

Git tag -> image digest -> rendered manifests -> staging smoke test -> GitOps promotion -> production verification -> rollback plan

This guide turns that path into a practical release runbook. It builds on CI/CD with GitHub Actions, CRD version upgrades and conversion webhooks, Helm-based Operator vs Flux vs Argo CD, and Debugging Kubernetes Operators.

Operator release pipeline in 60 seconds

• Treat the Git tag, manager image digest, CRD YAML, install manifests, Helm chart, and optional OLM bundle as one tested release unit.
• Build and publish the manager image from a semver tag, but promote by immutable digest.
• Render install artifacts from the same commit that produced the image.
• Deploy to staging first, apply a sample CustomResource, and wait for meaningful status conditions.
• Promote to production by changing GitOps state, not by running one-off kubectl set image commands.
• Roll back the whole release tuple when CRDs, conversion webhooks, storage versions, or finalizers changed.
• Publish OLM bundles only after the image digest and release metadata are known.

Release flow: Git tag to cluster

A practical operator release flow looks like this:

This structure matters because an operator release is an API release. A broken controller can usually be rolled back quickly. A broken CRD storage version or conversion webhook can trap existing resources in a much harder failure mode.

The release artifact contract

Every production release should answer one question: which exact artifacts were tested together?

If one artifact is owned by another team, document that ownership explicitly. For example, a platform team may own CRDs and webhooks while application teams own CustomResources. That can work, but only if the compatibility matrix and rollout order are clear.

Stage 1: Build, tag, and sign the manager image

Semver tags and immutable digests

Use semver tags such as v1.5.0 for releases. The tag should point to the commit that contains:

• controller source code,
• generated CRDs and RBAC,
• install manifests or chart changes,
• release notes or changelog entry,
• OLM bundle changes if the bundle is committed.

Build and push the manager image from that tag, then record the digest:

ghcr.io/example/database-operator@sha256:3d2f...

Human-readable tags such as v1.5.0 are useful, but production promotion should use the digest. A mutable tag can be repushed; a digest identifies the exact image content.

Registry layout

Use a predictable registry layout:

ghcr.io/example/database-operator:v1.5.0
ghcr.io/example/database-operator:v1.5
ghcr.io/example/database-operator:sha-a1b2c3d
ghcr.io/example/database-operator@sha256:...

Use per-commit tags for staging and semver tags for releases. Avoid latest in production manifests unless the cluster is intentionally disposable.

Signing, SBOM, and provenance

Add signing, SBOM, and provenance after the basic release path is reliable:

• sign the image digest with Cosign or Sigstore,
• publish an SBOM for the manager image,
• attach provenance from CI,
• verify signatures before promotion if your platform supports it.

These controls help consumers trust the digest, but they do not replace staging smoke tests.

Stage 2: Image to deployable manifests

Raw YAML, Helm, or Kustomize

Operator installs commonly use one of three packaging styles:

Kubebuilder's make deploy is a useful development starting point. Production installs usually add resource requests, Pod security settings, topology spread, affinity, image pull secrets, and tighter RBAC. See RBAC minimum permissions before broadening permissions.

Helm chart values

For a Helm-packaged operator, keep image settings explicit:

image:
  repository: ghcr.io/example/database-operator
  tag: v1.5.0
  digest: sha256:3d2f...

If both tag and digest are supported, production templates should prefer the digest. The tag remains useful for humans reading values files.

Kustomize image substitution

For Kustomize, CI or the GitOps promotion job can update the image:

kustomize edit set image controller=ghcr.io/example/database-operator@sha256:3d2f...

Commit that change to the environment repository instead of applying it directly to the cluster. The Git commit becomes the promotion audit trail.

Stage 3: Staging smoke test

Apply CRDs and webhooks before the manager

Order matters:

1. Apply CRDs.
2. Wait for CRDs to become established.
3. Apply RBAC and service accounts.
4. Ensure webhook service and certificates are ready if the operator uses webhooks.
5. Apply webhook configurations.
6. Roll out the manager Deployment.
7. Apply the sample CustomResource.

Useful checks:

kubectl wait --for=condition=Established crd/widgets.example.com --timeout=120s
kubectl rollout status deployment/database-operator-controller-manager -n database-operator-system --timeout=120s
kubectl get validatingwebhookconfiguration
kubectl get mutatingwebhookconfiguration

For GitOps, encode ordering with Argo CD sync waves or Flux dependencies when CRDs, webhooks, manager, and sample CRs are managed as separate units.

Apply a sample CustomResource

Use the smallest real CustomResource from your docs:

apiVersion: database.example.com/v1
kind: Database
metadata:
  name: smoke-test
spec:
  size: small

Then assert a meaningful condition:

kubectl wait database smoke-test --for=condition=Ready --timeout=180s

If your CRD does not expose a Ready condition, poll the exact status field your users rely on:

kubectl get database smoke-test -o jsonpath='{.status.phase}'

For condition design, see Kubernetes status and conditions.

Negative and cleanup checks

A release smoke test should prove more than "the Pod is running":

• Apply an invalid spec and confirm CRD validation or admission webhook rejects it.
• Delete the sample CR and confirm finalizers complete.
• Check operator logs for reconcile errors.
• Check Kubernetes events in the operator namespace.
• Confirm /healthz, /readyz, and metrics endpoints behave as expected.

Related deep dives: health probes, Prometheus metrics, and finalizers.

Stage 4: GitOps promotion

CI produces the digest; CD consumes it

CI should export a single fact:

IMAGE_DIGEST=ghcr.io/example/database-operator@sha256:3d2f...

CD or GitOps then consumes that digest by updating:

• a Helm values file,
• a Kustomize image patch,
• a raw manifest,
• an OLM bundle or catalog lane.

Avoid humans changing live Deployments without a matching Git change. If production drifts from Git, rollback and audit become much harder.

Argo CD and Flux ordering

Operator releases often need ordering beyond a normal app deployment.

For Argo CD, common ordering is:

For Flux, split CRDs, manager, and sample CRs into separate Kustomization resources when you need dependency ordering. Use dependsOn so the manager does not start before the API exists.

GitHub Environments and approvals

If GitHub Actions handles the promotion step, use GitHub Environments for production:

• required reviewers,
• environment-scoped secrets,
• deployment history,
• wait timers or change windows when needed.

Keep production registry credentials and cluster credentials out of pull request workflows.

Rollback matrix

The most common operator release mistake is assuming every rollback is only a Deployment rollback.

Treat the manager image, CRDs, webhooks, RBAC, Helm chart or Kustomize render, and OLM bundle as a tuple. Roll back one piece only when your compatibility matrix says it is safe.

CRD and controller versioning

Single release train

For most teams, one semver release should include:

• manager image,
• CRD YAML,
• RBAC and webhooks,
• Helm chart or Kustomize overlays,
• OLM bundle if used,
• changelog entry for breaking API changes.

This makes v1.5.0 mean "the exact API and controller we tested together."

Breaking vs non-breaking CRD changes

Non-breaking additions are usually easier:

• adding optional fields,
• adding enum values when clients tolerate them,
• adding status fields,
• adding printer columns.

Riskier changes need a migration plan:

• removing or renaming fields,
• changing validation in a way that rejects existing objects,
• changing storage versions,
• changing conversion webhooks,
• changing finalizer semantics.

For versioned APIs, follow the patterns in CRD version upgrades and conversion webhooks.

Safe rollout order

When conversion or webhook behavior changes, prefer this order:

1. Ensure the conversion webhook service can answer for old and new versions.
2. Apply compatible CRD changes.
3. Wait for CRDs to become established.
4. Roll out the new manager.
5. Verify existing CRs can still be listed, watched, and reconciled.
6. Only then promote to production.

The key question is not "did the Deployment roll out?" It is "can the Kubernetes API still serve every existing CustomResource correctly?"

OLM bundle and catalog publishing

OLM adds another release lane. The manager image and CRDs still come first; the bundle packages them for OLM users.

A typical OLM release path:

1. Build and push the manager image.
2. Generate the bundle with the final image reference.
3. Validate the bundle with a pinned operator-sdk version.
4. Build and push the bundle image.
5. Test install or upgrade from the bundle.
6. Update a catalog or file-based catalog.
7. Publish to internal catalog or submit to OperatorHub if applicable.

Common commands:

make bundle IMG=ghcr.io/example/database-operator@sha256:3d2f...
operator-sdk bundle validate ./bundle
make bundle-build BUNDLE_IMG=ghcr.io/example/database-operator-bundle:v1.5.0
make bundle-push BUNDLE_IMG=ghcr.io/example/database-operator-bundle:v1.5.0
operator-sdk run bundle ghcr.io/example/database-operator-bundle:v1.5.0

Keep bundle metadata in lockstep with the same image digest tested in staging. Bundle validation checks packaging; it does not replace runtime smoke tests.

For a deeper packaging guide, see OLM bundles & OperatorHub.

Pre-flight checks before production

Before promoting to production, answer these questions:

Checklist: gate, artifact, verification

See also

• CI/CD with GitHub Actions
• CRD version upgrades and conversion webhooks
• Operator capability levels (OLM)
• Helm-based Operator vs Flux vs Argo CD
• Debugging Kubernetes Operators
• Configuration: flags, env, live reload
• Pause and resume patterns
• RBAC minimum permissions

Upstream references

• Kubernetes CRD versioning
• Operator SDK packaging
• Argo CD sync phases and waves
• GitHub deployment environments
• OLM

Bottom line: release a Kubernetes Operator as a tested artifact set, not as a lone Deployment. Build the image once, promote the digest through Git, keep CRDs and webhooks compatible, prove the release with a real CustomResource in staging, and roll back the whole tuple when API shape or stored data is involved.